Methods and apparati for nondestructive detection of undissolved particles in a fluid

ABSTRACT

The apparati, methods, and computer program products disclosed herein can be used to nondestructively detect undissolved particles, such as glass flakes and/or protein aggregates, in a fluid in a vessel, such as, but not limited to, a fluid that contains a drug.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/241,861, which is a national stage application under 35 U.S.C. §371of International Application No. PCT/US2012/052914, filed Aug. 29, 2012,which in turn claims the benefit under 35 U.S.C. §119(e) to U.S.Provisional Application Nos. 61/691,211, 61/542,058 and 61/528,589,filed Aug. 20, 2012, Sep. 30, 2011 and Aug. 29, 2011, respectively, thecontent of each of which is incorporated by reference in its entiretyinto the present disclosure.

BACKGROUND

Differentiation between various types of particles is important in orderto characterize the quality of a given formulation of drug product. Forinstance, low specificity in differentiation has the potential toconfuse objects, such as glass lamellae, for proteinaceous particulatematter. High specificity of the differentiation system is needed inorder to provide accurate decisions when making decisions onformulations. Without information about the type(s) of particles in aparticular drug product, it may be difficult to formulate the drugproduct properly.

Unfortunately, conventional particle detection techniques are unsuitablefor detecting protein aggregates and other small and/or delicateparticles. Human inspectors usually cannot detect particles that aresmaller than about 100 microns. Automated inspection techniques aretypically destructive; that is, they involve removing the fluid beinginspected from its container, which usually renders the fluid unsuitablefor therapeutic use. In addition, conventional nondestructive inspectionsystems use only a single snapshot of the container to determine whetheror not particles are present, which often leads to imprecise particlesize measurements and/or particle counts. Conventional inspectiontechniques may also involve destruction of more delicate particles, suchas protein aggregates. For example, spinning a vial filled fluid at highspeed (e.g., 2000 rpm or more for several seconds) may rip apart proteinaggregates in the fluid.

SUMMARY

One embodiment of the technology disclosed herein relates to anapparatus for nondestructive detection of a particle (i.e., anundissolved particle) in a vessel that is at least partially filled witha fluid, such as an aqueous fluid, an emulsion, an oil, an organicsolvent. As used herein, the term “detection”, or “detecting”, is to beunderstood to include detecting, characterizing, differentiating,distinguishing, or identifying, the presence, number, location,identity, size, shape (e.g., elongation or circularity), color,fluorescence, contrast, absorbance, reflectance, or othercharacteristic, or a combination of two, three, four, five, six, seven,eight, nine, ten, eleven, twelve or more of these characteristics, ofthe particle. In illustrative embodiments, the apparatus includes animager to acquire time-series data representing a trajectory of theparticle in the fluid. A memory operably coupled to the imager storesthe time-series data, and a processor operably coupled to the memorydetects and/or identifies the particle. More specifically, the processorreverses a time ordering of the time-series data to form reversedtime-series data, estimates the trajectory of the particle from thereversed time-series data, and determines a presence or type of theparticle based on the trajectory. As defined herein reversed time-seriesdata includes frames of times-series data that have been arranged inreverse chronological order, such that the last-occurring event appearsfirst (and vice versa).

Further embodiments include a method and corresponding computer programproduct for nondestructive detection of an undissolved particle in avessel that is at least partially filled with a fluid. Implementing themethod involves reversing a time ordering of time-series datarepresenting a trajectory of the particle in the fluid to form reversedtime-series data, e.g., with a processor that executes instructionsencoded in a nonvolatile memory of the computer program product. Themethod further includes estimating the trajectory of the particle fromthe reversed time-series data, then detecting and/or identifying theparticle based on the trajectory.

Another embodiment is an apparatus for nondestructive detection of anundissolved particle in a vessel that is at least partially filled witha fluid, the apparatus which involves:

-   -   (a) at least two imagers positioned to image the particle from        different perspectives, each imager configured to acquire one or        more two dimensional images of the particle in the fluid;    -   (b) a memory operably coupled to the imager and configured to        store the time-series; and    -   (c) a processor operably coupled to the memory and configured to        detect the particle by:        -   (i) combining the two dimensional images from the at least            three imagers to determine three dimensional data indicative            of the position of the particle in the vessel; and        -   (ii) detecting the particle based at least in part on the            three dimensional data.

Also encompassed is a method for nondestructive detection of anundissolved particle in a vessel that is at least partially filled witha fluid, the method comprising:

-   -   (a) using at least two imagers to image the particle from        different perspectives to each acquire a respective one or more        two dimensional images of the particle in the fluid;    -   (b) combining the two dimensional images from the at least two        imagers to determine three dimensional data indicative of the        position of the particle in the vessel; and    -   (c) detecting the particle based at least in part on the three        dimensional data.

Other embodiments of the present invention include an apparatus, method,and computer program product for nondestructive detection of (one ormore) transparent or reflective objects (e.g., glass lamellae) in avessel that is at least partially filled with a fluid. An imageracquires data that represent light reflected from a plurality of spatiallocations in the vessel as a function of time and stores the data in amemory operably coupled to the imager. A processor operably coupled tothe memory detects the objects (e.g., glass lamellae), possibly inresponse to instructions encoded in the computer program product, basedon the data by identifying a respective maximum amount of reflectedlight for each location in the plurality of locations represented by thedata. The processor then determines a presence or absence of the objects(e.g., glass lamellae) in the vessel based on the number of spatiallocations whose respective maximum amount of reflected light exceeds apredetermined value.

Another embodiment of the invention is a method of nondestructivecounting and sizing of undissolved particles in a vessel that is atleast partially filled with a fluid. The method involves:

-   -   (a) receiving at least one image of the particles in the vessel        obtained under specified imaging conditions;    -   (b) based on the at least one image, detecting the particles and        determining information indicative of the apparent size of the        detected particles in the image;    -   (c) determining apparent particle size population information        indicative of an apparent particle size distribution of the        detected particles; and    -   (d) determining actual particle size population information        indicative of the actual particle size distribution of the        detected particles based on        -   (i) the apparent particle size population information and        -   (ii) calibration population information indicative of the            apparent size distribution of one or more sets of standard            sized particles imaged under conditions corresponding to the            specified imaging conditions.

Another embodiment of the invention is an apparatus for counting andsizing undissolved particles in a vessel that is at least partiallyfilled with a fluid, the apparatus including at least one processorconfigured to:

-   -   (a) receive at least one image of the particles in the vessel        obtained under specified imaging conditions;    -   (b) based on the at least one image, detect the particles and        determine information indicative of the apparent size of the        detected particles in the image;    -   (c) determine apparent particle size population information        indicative of an apparent particle size distribution of the        detected particles; and    -   (d) determine actual particle size population information        indicative of the actual particle size distribution of the        detected particles based on        -   (i) the apparent particle size population information and        -   (ii) calibration population information indicative of the            apparent size distribution of one or more sets of standard            sized particles imaged under conditions corresponding to the            specified imaging conditions.

A further embodiment of the invention is a computer program product fornondestructive counting and sizing of undissolved particles in a vesselthat is at least partially filled with a fluid, the computer programproduct comprising nonvolatile, machine-readable instructions, which,when executed by a processor, cause the processor to

-   -   (a) receive at least one image of the particles in the vessel        obtained under specified imaging conditions;    -   (b) based on the at least one image, detect the particles and        determine information indicative of the apparent size of the        detected particles in the image    -   (c) determine apparent particle size population information        indicative of an apparent particle size distribution of the        detected particles; and    -   (d) determine actual particle size population information        indicative of the actual particle size distribution of the        detected particles based on        -   (i) the apparent particle size population information and        -   (ii) calibration population information indicative of the            apparent size distribution of one or more sets of standard            sized particles imaged under conditions corresponding to the            specified imaging conditions.

A further embodiment of the invention is a method for nondestructivedetection of an undissolved particle in a vessel that is at leastpartially filled with a fluid, the method including:

-   -   (a) using at least one imager to image the particle;    -   (b) processing the image to determine position data indicative        of the position of the particle in the vessel;    -   (c) detecting the particle based at least in part on the        position data, where detecting the particle based at least in        part on position data includes identifying the presence of the        particle in a sub-region of the vessel;    -   (d) using a sensor to determine a characteristic of the particle        when the particle is located in the sub-region of the vessel,    -   (e) generating particle characteristic data indicative of the        determined characteristic; and    -   (f) associating the particle characteristic data with data        identifying the particle.

A further embodiment of the invention is an apparatus for nondestructivedetection of an undissolved particle in a vessel that is at leastpartially filled with a fluid, the apparatus including:

-   -   (a) at least one imager positioned image the particle;    -   (b) at least one sensor configured to determine a characteristic        of the particle when the particle is located in the sub-region        of the vessel;    -   (b) at least one processer operably couple to the each least one        imager and the sensor and configured to:        -   process the image to determine position data indicative of            the position of the particle in the vessel;        -   detect the particle based at least in part on the position            data and identify the presence of the particle in a            sub-region of the vessel;        -   use a signal from the sensor to determine a characteristic            of the particle when the particle is located in the            sub-region of the vessel,    -   generate particle characteristic data indicative of the        determined characteristic; and    -   associate the particle characteristic data with data identifying        the particle.

Another embodiment of the invention is an apparatus for nondestructivedetection of an undissolved particle in a vessel that is at leastpartially filled with a fluid, where the vessel includes a transparenttubular vessel wall disposed about a longitudinal axis, the apparatusincluding: an imager configured to acquire one or more images of theparticle in the fluid, the imager including a at least one imagingoptical element positioned to image the particle onto the sensor; anillumination source positioned at least partially within a plane passingthrough the vessel and substantially orthogonal to the longitudinal axisof the vessel, the illumination source arranged to substantiallyeliminate the presence of light rays emitted from the source thatreflect or refract from a surface of the vessel wall and are imaged bythe at least one optical element onto the sensor.

Another embodiment of the invention is a method for nondestructivedetection of an undissolved particle in a vessel that is at leastpartially filled with a fluid, wherein the vessel comprises atransparent tubular vessel wall disposed about a longitudinal axis, themethod comprising: using an imager to acquire one or more images of theparticle in the fluid, the imager comprising at least one imagingoptical element positioned to image the particle onto the sensor; andilluminating the vessel with an illumination source positioned at leastpartially within a plane passing through the vessel and substantiallyorthogonal to the longitudinal axis of the vessel, the illuminationsource arranged to substantially eliminate the presence of light raysemitted from the source that reflect or refract from a surface of thevessel wall and are imaged by the at least one optical element onto thesensor.

Unlike other particle detection systems and techniques, the inventivesystems and techniques operate nondestructively—there is no need toremove the fluid from the vessel to detect, count, and identify theparticles in the vessel. As a result, inventive systems and techniquescan be used to study changes in and interactions among the particles,the fluid, and the vessel over long time spans, e.g., minutes, hours,days, months, or years. In addition, inventive systems and techniques donot necessarily involve or result in the destruction of even moredelicate particles, such as small protein aggregates, in the vessel.They also capture time-series data, i.e., data representing thetrajectories of the particles in the moving fluid. Because the inventivesystems use time-series data instead of single-frame snapshots of thevessel, they can estimate more precisely the number of particles in thevessel and the particle sizes. They can also derive more informationabout each particle, such as particle morphology and particlecomposition, from the particle's motion. For example, falling particlestend to be denser than rising particles.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the following drawings and thedetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the disclosedtechnology and together with the description serve to explain principlesof the disclosed technology.

FIGS. 1A-1C show a visual inspection unit, a visual inspection imagingmodule, and a visual inspection platform, respectively, that can each beused to detect and identify particles in a container that is at leastpartially filled with a fluid.

FIG. 2A illustrates sample preparation, loading, and operation of thevisual inspection systems shown in FIGS. 1A-1C.

FIG. 2B shows processed images, captured by an illustrative visualinspection system, of particles and their trajectories in moving fluidin a vessel.

FIGS. 3A-3C illustrate three types of vessel agitation containing fluidand one or more particles in preparation from particle detection andidentification: rotation of a cylindrical vessel (FIG. 3A), inversionand rotation of a syringe (FIG. 3B), and rocking of a syringe (FIG. 3C).

FIG. 4 is a ray optics diagram of a telecentric lens used to image acylindrical vessel.

FIG. 5A shows the fluid meniscus and the recorded volume in acylindrical vessel containing fluid.

FIG. 5B illustrates distortion and blind spots in a cylindricalcontainer created by the container's shape.

FIGS. 5C and 5D illustrate techniques to compensate for distortion andblind spots when imaging cylindrical vessels.

FIG. 5E illustrates the distortion and blind spots in a cylindricalcontainer created by the container's shape for particles at variouspositions in the container.

FIG. 5F illustrates theoretical models for distortion caused by acylindrical container, each model corresponding to the same containerbut filled with a fluid with a different refractive index. The figurealso shows corresponding experimental measurements confirming thetheoretical models.

FIG. 5G illustrates the use of a corrective optical element to correctfor distortion in a cylindrical container, created by the container'sshape.

FIG. 5H is a detailed view of the corrective optical element of FIG. 5G.

FIG. 5I illustrates a device for selecting one of several correctiveoptical elements.

FIGS. 6A-6D show particle tracking systems with multiple imagers tocapture time-series data of the moving particles from many angles (FIGS.6A and 6B), at higher frame rates from the same angle (FIG. 6C), and atdifferent spatial resolutions from the same angle (FIG. 6D).

FIGS. 7A and 7B illustrate triggering of image acquisition andillumination for imaging particles with dual-sensor imagers.

FIG. 8 is a schematic diagram of an flexible, multipurpose illuminationconfiguration that includes light sources positioned before, behind, andbelow the vessel being inspected.

FIGS. 9A-9C illustrate illumination from different angles fordistinguishing between different particle species using the lightsources shown in FIG. 8.

FIG. 9D shows an illumination sequence and timing diagram for using theconfigurations of FIGS. 9A-9C to distinguish between different variousparticle species.

FIGS. 10A-10C illustrates glare from a vessel partially filled withfluid (FIG. 10A) and positioning of light sources outside a zone definedby revolving the imager about the vessel's longitudinal axis (FIGS. 10Band 10C).

FIGS. 10D-10E illustrate an alternative illumination scheme for reducingor eliminating glare from a vessel.

FIG. 11 is a schematic diagram of an imaging configuration suitable forimaging polarizing (e.g., chiral) particles.

FIG. 12 is a schematic diagram of an imaging configuration suitable forexciting and imaging fluorescent particles.

FIGS. 13A and 13B show maximum intensity projection images of glasslamellae (FIG. 13A) and protein (FIG. 13B) acquired with an illustrativevisual inspection system.

FIG. 14 includes flowcharts that illustrate different the overallparticle detection and identification process as well as imagepre-processing., particle tracking, and statistical analysissubprocesses.

FIGS. 15A and 15B show a frame of time-series data before (FIG. 15A) andafter (FIG. 15B) background subtraction.

FIG. 16A is a time-series data frame of a particle shown on eight-bitgrayscale (shown at left).

FIG. 16B is a close-up of the time-series data frame shown in FIG. 16B.

FIGS. 16C and 16D are thresholded versions of the time-series dataframes shown in FIGS. 16A and 16B, respectively.

FIGS. 17A-17D illustrate how a pair of successive frames of time-seriesdata (FIG. 17A) can used to perform predictive tracking (FIGS. 17B-17D).

FIG. 18A shows a grayscale time-series data frame showing severalparticles.

FIG. 18B shows a thresholded version of FIG. 18A used to locate aparticle's geometric center.

FIG. 19 shows successive time-series data frames that illustrateparticle collision/occlusion.

FIG. 20A shows a frame of time-series data with a pair or particles nextto each other inside a highlighted region.

FIGS. 20B-20E are successive frames of time-series data showing particleocclusion apparent as the particles in the highlighted region of FIG.20A propagate past each other.

FIGS. 21A-21C illustrate apparent occlusion of a moving particle causedby background subtraction of an artifact, such as a scratch or piece ofdirt, on a wall of the vessel for straight trajectories (FIG. 21A),curved trajectories (FIG. 21B), and parabolic trajectories (FIG. 21C).

FIGS. 22A-22C illustrate location of the center of mass of anirregularly-shaped particles using reversed time-series data (FIGS. 22Band 22C) and use of the center-of-mass location to determine particletrajectory (FIG. 22A).

FIGS. 23A-23D illustrate fluid dynamics observed and modeled incylindrical vessels. FIG. 23A shows changes in the shape of themeniscus. FIGS. 23B and 23C illustrate vortex formation inside afluid-filled vessel, and FIG. 23D shows particle trajectories in anillustrative vortex.

FIGS. 24A and 24B show close-ups of consecutive frames of reversedtime-series data where particle collisions have not been correctlyresolved (FIG. 24A) and the same plot after error correction (FIG. 24B).

FIGS. 25A-25E illustrate the time-dependence of particle sizemeasurement due to particle movement.

FIG. 25F is a graph of the time-dependent Feret diameter for theparticle shown in FIG. 25C.

FIG. 26A shows frames of processed time-series data at differentintervals with traces indicating different particle trajectories.

FIG. 26B shows illustrative measurements of a number of time-dependentparticle properties from particle trajectories in FIG. 26A.

FIGS. 27A-27F illustrate detection of a region of interest usingrear-angled illumination. FIG. 27A shows an original image (frame oftime-series data) that is subject to edge detection (FIG. 27B),grayscale thresholding (FIG. 27C), identification of the meniscus andvial-base (FIG. 27D), determination of a region of interest (bounded bydotted lines in FIG. 27E), and cropping (FIG. 27F) to yield an image ofthe fluid visible in the container.

FIGS. 28A-28C illustrate fill volume detection of a backlit vial. FIG.28A shows a raw image of the vial. FIG. 28B shows a region of interest(bounded by the dotted lines) determined using thresholding and edgedetection. Defects on the surface of the vial (shown in FIG. 28C) mayhamper fill volume detection.

FIGS. 29A-29D illustrate fill volume detection of a vial illuminate fromunderneath. FIGS. 29A and 29B are false-color images of a partially fullvessel (FIG. 29A) and an empty vessel (FIG. 29B). FIGS. 29C and 29Dillustrate automatic meniscus detection of partially full, empty, andpartially filled vessels.

FIG. 30 shows a processor suitable for processing time-series data.

FIGS. 31A-31D illustrate an example of grayscale thresholding for animage including a bright particle and a faint particle.

FIG. 32 shows a histogram of apparent particle sizes for a population ofparticles having a standard size (100 μm).

FIG. 33 shows apparent particle size count curves for two populations ofparticles, each population having the indicated standard size (μm).

FIG. 34 shows apparent particle size count calibration curves for fourpopulations of particles, each population having the indicated standardsize (μm).

FIG. 35 illustrates fitting a superposition of calibration curves to asample apparent particle size count curve.

FIG. 36 compares the results of two techniques of counting and sizingparticles, raw binning and LENS.

FIG. 37 illustrates a process for counting and sizing particlesfeaturing different sizing techniques for particles below and above athreshold size.

FIGS. 38A-38C illustrate particle tracking systems with multiple imagersto capture time-series data of the moving particles from multipleangles.

FIG. 39 illustrates the propagation of light rays through a containerreceived by each of two imagers (left panel) and each of three imagers(right panel) of the particle tracking systems of FIGS. 38A-C.

FIG. 40 shows particle detection results for an automated particledetection system (designated “APT”) compared with human results byvisual inspection.

FIG. 41 shows particle detection and classification results for anautomated particle detection system.

FIG. 42 shows a chart summarizing the linearity of particle count as afunction of sample dilution for and automated particle detection system.

FIG. 43 shows the precision of automated particle detection system usedto detect and count protein aggregate particles.

FIG. 44 shows protein aggregate particle detection results for anautomated particle detection system (designated “APT”) compared withhuman results by visual inspection.

FIG. 45 shows a spectrometer for use with a visual inspection unit.

DETAILED DESCRIPTION

FIG. 1A shows an exemplary automated visual inspection unit 100configured to non-destructively detect and/or identify particles in atransparent container 10 that is at least partially filled with fluid,such as a protein-containing pharmaceutical composition, drugs,biotechnology products, drinks, and other translucent fluids regulatedby the U.S. Food and Drug Administration.

Although detection of the presence or absence of a particle can beaccomplished by viewing portions of the container in which the exterioris non-uniform (e.g. the heel), in typical embodiments, for particlecharacterization measurements such as counting and sizing, it may benecessary to look at the particles through the substantially uniformvertical wall of the container in order to mitigate distortions. Thishas implications on minimum fill volume, as the apparent two dimensionalcross section of the fluid in the container 10 visible to the unit 100must be of an appropriate area to provide usable statistics. Therequired fill volume is dependent on the circular diameter of thecontainer (smaller containers, less fill volume required). In variousembodiments, the interior volume of the container may be at least 1%, atleast 5%, at least 10%, at least 20%, at least 30%, at least 40%, atleast 50%, at least 60%, at least 70%, at least 80%, at least 90%, or atleast 100% filled with fluid.

In various embodiments, the particle detection techniques describedherein are optical in nature. Accordingly, in some embodiments, thewalls of container 10 are sufficiently transparent at the illuminatingwavelength to allow visualization of the liquid contained within. Forexample, in some embodiments, the container 10 may be made from clearborosilicate glass, although other suitable materials may be used. Theturbidity of the fluid contained within the vessel is also ofimportance, and should be sufficiently low to allow for the desiredlevel of visualization. In some embodiments, embodiments, the fluid hasturbidity in the range 0-100 NTU (Nephelometric Turbidity Units),preferably 0-20 NTU, and more preferably 0-10 NTU. Standard practicesfor turbidity measurement may be found, e.g., EPA Guidance Manual,Turbity Provisions, Chapter 3 (April, 1999).

Illustrative systems can detect and identify transparent and/ortranslucent particles that refract and/or scatter light (e.g., proteinaggregates, glass flakes or lamellae, and blobs of oil), particles thatreflect light (e.g., metal flakes), and/or particles that absorb light(e.g., black carbon and plastic particles) based on their differentoptical characteristics. Some inventive visual inspection units 100 candetect all three classes of particle by using illumination sequencessuch as those described below. Inventive visual inspection units 100 mayalso be specially configured to detect, identify, and/or track proteins,which may appear as densely bound aggregates, loosely bound cotton woolsubstances with high water content, (reflective) crystals, gelatinoussubstances, and/or amorphous aggregates.

The term “protein,” which may be used interchangeably with the term“polypeptide,” refers in its broadest sense to a compound of two or moresubunit amino acids, amino acid analogs or peptidomimetics. The subunitsmay be linked by peptide bonds. In another embodiment, the subunits maybe linked by other bonds, e.g., ester, ether, etc. As used herein theterm “amino acid” refers to natural and/or unnatural or synthetic aminoacids, including glycine and both the D and L optical isomers, aminoacid analogs and peptidomimetics. A peptide of three or more amino acidsis commonly called an oligopeptide if the peptide chain is short. If thepeptide chain is long, the peptide is commonly called a polypeptide or aprotein. The term “peptide fragment” as used herein, also refers to apeptide chain.

The container 10 may be a rectangular or cylindrical vessel made ofglass or plastic (e.g., a cuvette, vial, ampoule, cartridge, test tube,or syringe); it can also have another shape and/or be made of differentmaterial, so long as it provides visualization of the container contentsat the imaging wavelength. Although particular embodiments provide clearand unperturbed visualization of the container contents, otherembodiments may time image acquisition to coincide with periods when thecontainer is unperturbed and/or employ postprocessing to compensate fordistortion of the recorded data.

The unit 100 includes an imager 110 with collection optics that projectimages of the container contents onto a sensor. In this case, thecollection optics include a telecentric lens 114, and the sensor is acharge-coupled device (CCD) 112. Memory 140 coupled to the CCD 112records and stores a stream of images representing the containercontents, and a processor 130 coupled to the memory 140 analyzes therecorded image sequence as described below to detect and identify theparticles in the container 10. As understood by those of skill in theart, the processor 130 may be implemented with a suitably configuredgeneral-purpose computer (e.g., one using an Intel® Core™ i5 or AdvancedMicro Devices Athlon™ processor), field-programmable gate array (e.g.,an Altera® Stratix® or Xilinx® Spartan®-6 FPGA), or application-specificintegrated circuit. The memory 140 may be implemented in solid-statememory (e.g., flash memory), optical disc (e.g., CD or DVD), or magneticmedia, and can be selected to be any appropriate size (e.g., 1 GB, 10GB, 100 GB, or more).

An illumination system 120, which includes one or more light sources 122a and 122 b disposed around the container 10, illuminates the container10 and its contents during image acquisition. The visual inspection unit100 can be integrated into an inspection module 160 that also includes aspindle 150, shaker, ultrasonic vibrator, or other agitator to spin,shake, or otherwise agitate the container contents prior to imaging andto hold the container 10 during imaging, as in FIG. 1(b).

FIG. 1(c) shows a medium-to-high throughput visual inspection platform170 that includes one or more inspection modules 160-1 through 160-5(generally, inspection modules 160), a robot 180, and a vial tray 172,which holds uninspected and/or inspected containers 10 in individualcontainer wells. Upon instructions from a user or automatic controller(not shown), the robot 180 moves a container 10 from the vial tray 172to an inspection module 160, which captures and records time-series dataof particles moving the container 10. The robot 180 then returns thecontainer 10 to the vial tray 172.

In some examples, the top layer of the vial tray 172 and/or rims of thecontainer wells are made of Delrin® acetal resin or another similarmaterial, and the interior edges of the container wells are beveled toprevent the containers 10 from becoming scratched as they are insertedinto and removed from the container wells. The vial tray 172 may includea base layer made of aluminum or another similar material that does noteasily warp or crack. The walls of the container wells are typicallythick to hold the vials securely as the tray 172 is carried (e.g., by aperson) to and from the visual inspection platform 170. Depending on itsconstruction, the vial tray 170 may hold the containers 10 in predefinedpositions to within micron-scale tolerances to facilitate containerretrieval and insertion by the robot 180, which may operate withmicron-scale precision.

The robot 180 is a “pick-and-place” system that plucks vials from thetray 172, moves each container 10 along a rail 182 that extends fromabove the tray 172 to above the spindles 160, and places the container10 on a particular spindle 160. Some robots may also be configured tospin the container 10 before placing the container 10, obviating theneed for a spindle 160. Alternatively, the robot 180 may include asix-axis robotic arm that can spin, vibrate, and/or shake (e.g., performthe “back-and-forth” needle shaking described below) the container 10,which also obviates the need for spindles 160. Those of skill in willreadily appreciate that other loading and agitation mechanisms andsequences can be used with the inventive visual inspection systems andprocesses.

The visual inspection platform 170 operates as shown in FIG. 2(a). Instep 202, the containers 10 to be inspected are cleaned (e.g., by handusing appropriate solvents), then loaded into the tray 172 in step 204.The robot 180 extracts a container 10 from the tray 172 and places it onthe spindle 160. Next, in step 206, the processor 130 determines thesize and location of the meniscus and/or region of interest (ROI),(e.g., the portion of the container 10 filled with fluid), from an imageof the static container 10 acquired by the imager 110. Alternatively,the user can specify the location of the meniscus and/or the region ofinterest if the fill volume and container shape and volume are knownwith sufficient certainty. Once the processor 130 has located the ROI,the spindle 160 spins and stops the container 10 in step 208, whichcauses the fluid to move and particles in the container 10 to becomesuspended in the moving fluid. In step 210, the imager 110 recordstimes-series data in memory 140 in the form of a sequence of staticimages (called “frames”) representing snapshots of the ROI, taken atregularly spaced time intervals.

After the imager 110 has acquired enough time-series data, the processor130 subtracts background data, which may represent dirt and/or scratcheson one or more of the surfaces of the container. It may also filternoise from the time-series data as understood by those of skill in theart and perform intensity thresholding as described below. The processor130 also reverses the ordering of the time-series data. That is, if eachframe in the time-series data has an index 1, 2, . . . , n−1, n thatindicates the order in which it was acquired, the frames in the reversedtime-series data are arranged with indices ordered n, n−1, . . . , 2, 1.If necessary, the processor 130 also selects start and end points of thedata to be analyzed as described below. (Those of skill in the art willreadily appreciate that the processor 130 may perform backgroundsubtraction, noise filtering, intensity thresholding, time-series datareversal, and start/end point determination in any order.) The processor130 tracks particles moving in or with the fluid in step 212, thensizes, counts, and/or otherwise characterizes the particles based ontheir trajectories in step 214.

Each inspection module 160 may perform the same type of inspection,allowing for parallel processing of the containers 10; the number ofmodules 160 can be adjusted depending on the desired throughput. Inother embodiments, each module 160 may be configured to performdifferent types of inspections. For example, each module 160 may inspectparticles at a different illumination wavelength: module 160-1 may lookfor particles that respond to visible light (i.e., radiation at awavelength of about 390 nm to about 760 nm), module 160-2 may inspectcontainers using near infrared illumination (760-1400 nm), module 160-2may inspect containers using short-wavelength infrared illumination(1.4-3.0 μm), module 160-4 may inspect particles at ultravioletwavelengths (10-390 nm), and module 160-5 may inspect particles a X-raywavelengths (under 10 nm). Alternatively, one or more modules 160 maylook for polarization effects and/or particle fluorescence.

In embodiments with different types of modules 160, the first module160-1 may perform preliminary inspections, with subsequent inspectionscontingent upon results of the preliminary inspections. For instance,the first module 160-1 may perform a visible-light inspection thatsuggests that a particular container contains polarization-sensitiveparticles. The processor 130 may then instruct module 160-2, which isconfigured to perform polarization-based measurements, to inspect thecontainer in order confirm (or disprove) the presence ofpolarization-sensitive particles. Visible-light time-series dataacquired by module 160-1 may indicate the presence of several particlesin a particular container 10, but not the particle type, which may leadthe processor 130 to order, e.g., infrared inspection at module 160-3.

Container Agitation to Induce Particle Movement

As described above, mechanically agitating the container 10 causesparticles at the bottom of the container 10 or on the sides of thecontainer's inner walls to become suspended in the fluid within thecontainer. In particular embodiments, the user and/or visual inspectionsystem selects and performs an agitation sequence that causes the fluidin the container to enter a laminar flow regime, which is regime inwhich the fluid flows in parallel layers, with no eddies, swirls, ordisruptions between the layers. In fluid dynamics, laminar flow is aflow regime characterized by high momentum diffusion and low momentumconvection—in other words, laminar flow is the opposite of rough,turbulent flow. Agitation also causes the particles to become suspendedin the moving fluid. Eventually, friction causes the fluid to stopmoving, at which point the particles may stick to the walls of thecontainer or settle to the bottom of the container.

Compared to turbulent flow, laminar flow yields smoother particlemotion, which makes it easier to estimate particle trajectories. (Ofcourse, the processor may also be configured to estimate particletrajectories in certain turbulent flow regimes as well, provided thatthe sensor frame rate is fast enough to capture “smooth” sections of theparticle trajectories.) If desired, the container can be agitated inmanner that produces substantially laminar flow. For example, a spindlemay rotate the container at a specific velocity (or velocity profile)for a specific time as determined from measurements of fluid behaviorfor different container sizes and shapes and/or different fluid levelsand viscosities.

In one particular embodiment, a servo or stepper motor drives a spindlethat holds a cylindrical container, causing the container to spin aroundits central axis, as shown in FIG. 3(a). Spinning the container 10 atsufficient speed causes even heavy particles (such as metal flakes) torise from the bottom of the container 10 and into the fluid. For manyfluids and particles, the motor drives a spindle holding the container10 at 300 rpm for about three seconds. (Higher spin speeds may berequired to energize heavy particles.) After three seconds of spin, themotor stops abruptly, and the fluid is allowed to flow freely in thenow-static container. At this point, the imager 110 begins capturingvideo of the rotating fluid. The memory 140 records video for up toabout seven to fifteen seconds, depending on the size of container underscrutiny (the memory 140 records less video of fluid in smallercontainers because the fluid slows down more quickly in smallercontainers due to the increased impact of wall drag).

In another embodiment, the spindle rotates the container 10 in atwo-phase agitation/imaging sequence. In the first phase, the spindlespins the container 10 at 300 rpm for three seconds, causing less dense(and more delicate) particles, such as proteins, to become suspended inmoving fluid. The imager 110 then captures video of proteins in themoving fluid. Once the imager 110 has collected enough time-series data,the second phase begins: the spindle rotates the container 10 at about1600-1800 rpm for one to three seconds, causing denser particles, suchas metal flakes, to become suspended in moving fluid, and the imager 110captures time-series data representing the denser particles moving inthe container 10. The high-speed rotation in the second phase may beintense enough to temporarily dissolve or denature the proteinaggregates, which can re-form after the fluid slows or stops moving. Thetwo-phase operation makes it possible to detect both dense particlesthat may not be energized by low-speed rotation and proteins that may bedenatured by high-speed rotation.

Inventive systems may employ other rotation sequences as well, dependingon (but not limited to) any of the following parameters: fluidviscosity, fluid fill level, fluid type, surface tension, containershape, container size, container material, container texture, particlesize(s), particle shape(s), particle type(s), and particle density. Forexample, inventive systems may spin larger containers for longer periodsof time before imaging the container contents. The exact agitationprofile for a given fluid/container combination can be computed,characterized, and/or determined by routine experimentation.

If the visual inspection module uses a predetermined agitation sequencefor a well-characterized container/fluid combination, it may triggerdata acquisition only when the fluid (and suspended particles) are in alaminar flow regime. Alternatively, it may acquire additionaltime-series data, and the processor may automatically select start andend frames based on the container/fluid combination and/or agitationsequence.

In some embodiments, data acquisition may be triggered based on adetected characteristic of the fluid flow in the container. For example,as described in detail below, in some embodiments, it is possible todetect the meniscus of the fluid in the container and monitor themovement of the meniscus to determine when a vortex in the fluid relaxespost-spin. In some such cases the data acquisition may begin when thedetected movement of the meniscus has returned to a substantially stablestate.

Any of the visual inspection systems described above can also be used todetect and/or identify native and foreign particles in a syringe 12 thatis at least partially filled with a drug product 32 or other fluid, asshown in FIG. 3B. Syringes 12 are often stored needle-down. As such,particulate may settle in the syringe's needle 34. To visualize theseparticles, a robot or person inverts the syringe 12—i.e., the robot orperson rotates the syringe 12 by 180° about an axis perpendicular to itslongitudinal axis so that the needle 34 points up. Particulate that hassettled in the needle 34 falls vertically, enabling visualization by theimager 110. The robot or person may also spin syringe during the flip tofully mobilize the fluid.

Many syringes 12 have barrels with relatively small inner diameters(e.g., about 5 mm), which dramatically increases the effect of walldrag. For many drug products 32, the wall drag causes all rotationalfluid motion to cease within approximately one second. This is a veryshort time window for practical particle analysis. Fortunately, rockingthe syringe 12 gently about an axis perpendicular to its longitudinalaxis, as shown in FIG. 3(c), yields particle motion that lasts longerthan one second. The lateral rocking, which can be done with a robot orby hand, agitates particles through the motion of the syringe 12 and themotion of any air bubble(s) 30 oscillating within the barrel of thesyringe 12. The visual inspection modules, units, and platformsdescribed above are designed to be reconfigurable, and can accommodatethis alternative method of agitation.

Once agitation is complete, the visual inspection system should remainstill for the video recording phase. Because of the high resolution ofthe imagers typically employed, the spatial resolution of the images isvery fine (e.g., about ten microns or less) and can be at least as fineas the diffraction limit. For certain configurations, a small (e.g.,ten-micron) movement of the sample equates to a full pixel of movementin the detected image. Such motion compromises the effectiveness ofstatic feature removal (background subtraction), which in turn degradesthe performance of the analysis tools and the integrity of the outputdata.

With this in mind, vibration isolation is a key design consideration. Inparticular embodiments, the base of an illustrative visual inspectionsystem is mechanically isolated from the laboratory environment, e.g.,using vibration-dampening shocks, floats, and/or gaskets. Additionally,inside the unit, sources of vibration such as computers and robotcontrollers can be mechanically isolated from the rest of the system.Alternatively, data acquisition can be synchronized with residual motionof the container with respect to the imager or performed with a camerathat performs pixel shifting or some other motion-compensating behavior.Such residual motion can also be recorded for postprocessing to removedeleterious effects of image motion.

Imager Configurations

Illustrative visual inspection systems can use standard, off-the-shelfimagers with any suitable sensor, including, but not limited to chargecoupled device (CCD) or complementary metal-oxide-semiconductor (CMOS)arrays. The choice of sensor is flexible and depends somewhat on therequirements of the particular application. For instance, sensors withhigh frame rates enable the accurate mapping of the trajectories offast-moving particles (e.g., in low-viscosity fluids). Sensitivity andnoise performance are also important because many protein particles aretransparent in solution and scatter light weakly, producing faintimages. To improve noise performance, the sensor can be cooled, asunderstood in the art. For most applications, monochrome sensors offerthe best performance due to slightly higher resolution over colorcameras, as well as boasting higher sensitivity. For a small subset ofapplications, however, color sensors may be preferred because theycapture the color of the particle, which may be very important inestablishing its source (e.g., clothing fiber). In product qualityinvestigation (also known as forensics), for instance, color sensors canbe useful for distinguishing between different types of materials (e.g.,fibers) in the manufacturing facility that can contaminate the drugproduct.

For complete container inspection, the imager's field of view shouldencompass the whole fluid volume. At the same time, the imager should beable to resolve small particles. Visual inspection systems achieve bothlarge fields-of-view and fine resolution with large-format,high-resolution sensors, such as the Allied Vision Technologies (AVT)Prosilica GX3300 eight-megapixel CCD sensor, which has 3296×2472 pixels.Other suitable sensors include the ACT Pike F505-B and Basler PilotpiA2400-17gm five-megapixel cameras. When the imaging optics are chosento fully image the fluid-bearing body of a 1 ml BD Hypak syringe, theAVT Prosilica GX3300 CCD sensor captures time-series data with a spatialresolution of approximately ten microns per pixel in both transversedimensions. The combination of high speed and high resolution impliesthat recording the time-series data may involve large data transferrates and large file sizes. As a corollary, the video compressiontechniques described below are specially designed to reduce data storagerequirements while preserving the integrity of the fine detail of theparticles captured in the image.

The collection optics that image the region of interest onto the sensorshould be selected to provide a sharp images of the entire volume with aminimum spot size that is equal to or smaller than the pixel size of thesensor to ensure that the system operates with the finest possibleresolution. In addition, the collection optics preferably have adepth-of-field large enough to fit the entire sample volume.

Telecentric lenses, such as the lens 114 shown in FIG. 4, are especiallywell-suited to visual inspection of fluid volumes because they arespecifically designed to be insensitive to depth of field. As understoodby those of skill in the art, a telecentric lens is a multi-element lensin which the chief rays are collimated and parallel to the optical axisin image and/or object space, which results in constant magnificationregardless of image and/or object location. In other words, for anobject within a certain range of distances from an imager with atelecentric lens, the image of the object captured by the imager issharp and of constant magnification regardless of the object's distancefrom the imager. This makes it possible to captures images in whichparticles at the ‘back’ of the container 10 appear similar to those atthe ‘front’ of the container 10. The use of a telecentric lens alsoreduces the detection of ambient light, provided a uniform darkbackplane is used. Suitable telecentric lenses 114 include the EdmundOptics NT62-901 Large Format Telecentric Lens and the Edmund OpticsNT56-675 TECHSPEC Silver Series 0.16× Telecentric Lens.

Container-Specific Blind Spots

One goal for almost any visual inspection system is to provide 100%container volume inspection. In reality, however, there may be fixedzones in which particles cannot be detected, as shown in FIG. 5A. First,the liquid around the meniscus may be difficult to incorporate in theanalysis because the meniscus itself scatters light in a manner thatpotentially saturates the detector at that location, obscuring anyparticles or other features of interest. Second, for vials, the base ofthe container is typically curved at the corner, generally referred toas the ‘heel’. The curved heel has the effect of distorting andultimately obscuring any particles that venture sufficiently close tothe bottom of the vial. Third, for syringes, the rubber plug features acentral cone which intrudes slightly into the container volume. The tipof this cone can potentially hide particles, although it is small. Themost subtle blind spots occur due to the curvature of the vial.

Cylindrical containers may also cause a lensing effect, shown in FIG.5B, (indicated by bent rays 18) which serves to undermine theperformance of the telecentric lens. The container's curved walls alsocreate blind spots 14.

FIG. 5E shows an example of the lensing effect cause by a cylindricalcontainer 10. The camera/observer is at the bottom of the figure. Asdescribed above, a telecentric lens may be used when imaging particlesin the container 10 to ensure that particles have a consistentappearance in the image that does not depend on their position in thecontainer, that is, their distance from the camera. To accomplish this,in some embodiments, the depth of focus of the telecentric lens ischosen to be larger than the diameter of the fluid volume. In someembodiments, in the absence of a corrective optical element, thecontainer curvature undermines this principle.

As shown, the shape and magnification of a imaged particle in thecontainer 10 will depend on the position of the particle in thecontainer. A particle 501 at the front-and-center of the container isnot distorted at all (top inset). An identical particle 502 at therear-and-side is distorted the most (bottom inset). Note that for acylindrical container, the distortion occurs only along the horizontalaxis (as is evident in the bottom inset).

To mitigate these effects, optional corrective optics, such as acorrective lens 116, are placed between the telecentric lens 114 and thecontainer 10 as shown in FIG. 5C. Additional spatial correction optics118 may provide additional compensation for distortion caused by thecontainer's shape as shown in FIG. 5D. In various embodiments, anysuitable corrective optical elements, e.g., tailored based on thecurvature of the container 10 and/or the refractive index of the fluid,may be used in addition or alternative to the corrective lens 116 andoptics 118.

For example, in some embodiments, a model of the lensing effect causedby the cylindrical container 10 may be developed. The model may be basedon a suitable set of parameters characterizing the optical distortionincluding, for example, the container outer diameter, container innerdiameter, container refractive index, liquid refractive index, andwavelength of illumination light. The model may be developed using anysuitable techniques know in the art including, for example, ray tracingtechniques. FIG. 5F shows examples of theoretical models for the lensingeffect for two different sets of container parameters (top left, bottomleft), along with experimental data for the corresponding physicalsituations (top right, bottom right). As shown, the theoretical modeland experimental data are in excellent agreement.

Referring to FIGS. 5G and 5H, a corrective optical element 503 (as showna lens) is used to correct for the lensing effect described above. Thedesign of the corrective optical element may be based on a theoreticaloptical model of the container, experimental data indicative of theoptical properties of the container, or combinations thereof. As shown,the corrective optical element 503 is made of a refractive materialhaving cylindrical front and back surfaces. In some embodiments thedesign of the lens may be determined using free parameters including theradius of the front and back surfaces, the thickness of the lens, therefractive index of the lens, and the position of the lens relative tothe container.

In some embodiments, other shapes can be used for the front and backsurfaces of the lens, e.g., parabolic or arbitrary custom shapes. Insome embodiments, relaxing the requirement that the surfaces becylindrical will increase the size of the parameter space for the designof the corrective optical element 503 thereby allowing improvedcorrection.

In some embodiments, the corrective optical element 503 may includemultiple elements, thereby further increasing the design parameterspace. In some embodiments, the corrective optical element 503 maycorrect for other types of optical distortion, aberration, or othereffects. For example, in cases where illumination at multiplewavelengths is used, the corrective optical element 503 may be used tocorrect for chromatic aberration.

In some embodiments, the corrective optical element 503 may be designedto correct for distortion caused by a particular container and/or fluidtype. Because a single automated visual inspection unit 100 may be usedwith multiple container types, in some embodiments, it may be desirableto allow the corrective optical element 503 to be selectably changed tomatch the specific container 10 under inspection. For example, FIG. SIshows a rack 504 that holds multiple corrective optical elements 503.The rack may be moved (manually or automatically) to place a selectedone of the elements into the optical chain for an imager 110. Note thatalthough a rack is shown, in various embodiments, any other suitablemechanism for selecting one optical element out of a set of multipleoptical elements may be used.

Alternative visual inspection systems may include adaptive optics tocompensate for distortion due to the container's curvature. For example,the telecentric lens 114 may be configured to capture an image of thecontainer 10 reflected from a deformable mirror, such as amicro-electrical-mechanical system (MEMS) mirror. The sensor 112 usesthe background data to derive the nature and magnitude of theaberrations resulting from surface curvature, surface defects, and otherimperfections in the container 10. The sensor 112 feeds this informationback to the deformable mirror, which responds by adjusting its surfaceto compensate for the aberrations. For example, the deformable mirrormay bend or curve in one direction to compensate for the containercurvature. Because the deformable mirror responds dynamically, it can beused to compensate for aberrations specific to each individual container10.

In addition, particle tracking can be tuned to detect particledisappearance in conjunction with the known locations of these blindspots, allowing the program to predict if and where the same particlemight re-appear later in the video sequence as described below.

Additional techniques for dealing with blind spot related issues (e.g.,using multiple imagers) are described below.

Camera Frame Rate

Effective particle tracking using the nearest-match (greedy) algorithmdescribed below can be considered as a function of three primaryfactors: the camera capture rate (frame rate), the particle density (inthe two-dimensional image) and the typical particle velocity. For trulyeffective tracking using the nearest-match algorithm, the camera shouldpreferably be fast enough to meet the criterion:

${{Camera}\mspace{14mu} {rate}} > {\frac{{Maximum}\mspace{14mu} {particle}\mspace{14mu} {velocity}}{{Minimum}\mspace{14mu} {interparticle}\mspace{14mu} {separation}\mspace{14mu} {distance}}.}$

In reality, when projecting a three-dimensional volume onto a twodimensional image, it is possible for particles to appear to be veryclose to one another (even occluding one another) when in fact they arewell spaced in the container. When taking this into account, it makesmore sense to consider the mean nearest-neighbor distance than toconsider the apparent minimum interparticle separation distance. Notethat here that nearest-neighbor distance is the distance betweenadjacent particles in a given frame of time-series data, whilenearest-match distance refers to the distance between the difference inposition observed for a single particle in consecutive frames oftime-series data. Rewriting the criterion for camera speed in terms ofnearest-match distance gives:

${{Camera}\mspace{14mu} {rate}} > {\frac{{Maximum}\mspace{14mu} {particle}\mspace{14mu} {velocity}}{{Minimum}\mspace{14mu} {interparticle}\mspace{14mu} {separation}\mspace{14mu} {distance}}.}$

Alternative visual inspection systems may use predictive trackingtechniques instead of nearest-match (greedy) particle trackingtechniques. Predictive techniques use knowledge of a particle's knowntrajectory, in conjunction with knowledge of the spatial constraints ofthe container and the expected fluid behavior, to make estimate theparticle's most likely position in a subsequent frame. When properlyimplemented this approach can more accurately track particles movingthrough densely populated images at speed.

When attempting to detect and measure very small particles in relativelylarge containers, it is advantageous to maximize the spatial resolutionof the image sensor. In general, this has the direct effect of loweringthe sensor's maximum achievable frame rate.

Visual Inspection with Multiple Imagers

The use of a single camera may compromised by the presence of knownblind spots. Additionally, mapping a three-dimensional particledistribution onto a two-dimensional image can result in ambiguity due toocclusion (e.g., as shown in FIG. 5E, where a particle at the backcenter of the container is occluded by a particle at the front center).Alternative visual inspection systems (e.g., as seen in FIG. 6) can, inprinciple, resolve this problem by correlating results from two or moreimaging systems. By correlating positional trajectory information fromtwo or more cameras it is possible to construct detailedthree-dimensional trajectory maps, which may be more robust and lessprone to errors caused by occlusion (discussed below) thantwo-dimensional trajectory maps.

Increasing the spatial resolution of the imager also limits the dataacquisition rate (frame rate) for a given particle concentration andparticle speed. When inspecting unknown containers, there can be noguarantee the particle concentration will be suitably low. At the sametime, in order to suspend heavy particles such as glass or metal in thefluid, rotation rates in the container may need to be quite high,resulting in high particle velocities in the captured video stream. Oneway to resolve this conflict is to employ the novel imaging hardwareconfigurations described below. Assuming the best commercially availablesensors are already being employed, and the particles in the containerare scattering a sufficient amount of light, it is still possible toincrease the data acquisition rate by multiplexing two or more sensors,with constant, reliable triggering from a dedicated trigger source.

In addition, exemplary visual inspection systems can be configured toprovide spatial resolution finer than 10 microns by relaxing therequirement for full container inspection, and instead consider only asubset of the volume. In general, for sub-visible particles, especiallyprotein aggregates, this is acceptable because smaller particles tend tooccur in higher numbers and be more homogenously distributed throughoutthe volume. Alternatively, exemplary visual inspection systems canprovide both full container inspection and fine spatial resolution byusing multiple imagers with different magnifications to acquire bothwide-area and fine-resolution time-series data in parallel.

Alternative magnifications can be used simultaneously, e.g., as in FIG.6A, with one imager 1102 to look at the full container, and a secondimager 1104 with higher magnification (e.g., a long-working distancemicroscope objective) to zoom in on a smaller sub-volume and examine,for instance, very small particles (e.g., particles with diameters ofabout ten microns, five microns, one micron or less). Other visualinspection systems may include multiple imagers 1102, 1104, and 1106disposed about a container 10 illuminated by one or more rings oflight-emitting diodes (LEDs) 1120 mounted above and below the container10 as shown in FIG. 6B. Identical imagers 1102 mounted at differentposition provide binocular vision. An imager 1104 with along-working-distance microscope objective provides fine resolution fora subvolume of the container 10, and an imager 1106 with an alternativesensor (e.g., an infrared sensor, bolometer, etc.) provides additionaltime-series data.

FIGS. 6C and 6D show alternative imaging configurations that harness theproperties of telecentric imaging. At the back aperture of thetelecentric lens, a 50/50 beamsplitting cube 1202 splits the projectedimage into two separate imaging arms. Each imaging arm may include ahigh-resolution, low-speed sensor 1222 that operates in interleavedfashion with the sensor 1222 in the other arm as shown in FIG. 6C todouble the frame rate. That is, running the two sensors 1222simultaneously with a half-cycle relative phase offset improves temporalresolution by a factor of two. The image streams can then be combined toprovide a single movie at double the nominal sensor frame rate.

Alternatively, each arm may include a different sensor as shown in FIG.6D, e.g., to compensate for a tradeoff in imaging sensor arrays: thefiner the camera resolution, the slower the camera's maximum possibleframe rate (e.g., 10-50 or 15-25 frames per second at full resolution,50-200 frames per second at low resolution, etc.). For accurate particletracking, the dominant sensor performance parameter is high temporalresolution (high frame rate). For accurate particle sizing, however, thedominant sensor performance parameter is fine spatial resolution (asmany pixels as possible in the image). At present, the primary limitingfactor on the spatial resolution and data transfer rate is the datatransfer bus. Available imagers can acquire time-series data of afour-centimeter tall container with a spatial resolution of about tenmicrons per pixel and a data transfer rate of about twenty-five framesper second for a standard personal computer bus (e.g., a dual GigE orCameraLink bus).

FIG. 6D illustrates one way to achieve fast frame rates and fineresolution: image the fluid with both a high-resolution, low-speedsensor 1222, and a sensor 1224 with a more modest spatial resolution,but a higher frame rate. External triggering can ensure the two camerasare synchronized in a commensurate fashion. Because the cameras areviewing copies of the same image, their data can be directly correlatedto produce improved particle analysis.

FIGS. 7A and 7B illustrate timing and control of illumination sources120 and multiple cameras. In both FIG. 7A and FIG. 7B, a triggercontroller 702 emits two trigger signals—labeled ARM 1 and ARM 2 inFIGS. 7A and 7B—derived by decimating a master pulse signal. The ARM 1trigger signal drives a first camera (1102 a in FIG. 7A, 1222 a in FIG.7B) and the ARM 2 trigger signal drives a second camera (1102 b in FIG.7A, 1222 b in FIG. 7B) in interleaved fashion. That is, the triggersignals causes the first and second cameras to acquire alternatingsequences of frames. The trigger controller 702 may also drive theillumination source 120 with an illumination signal that causes theillumination source 120 to illuminate the container every time the firstor second camera acquires an image. Other trigger sequences are alsopossible; for example, the trigger controller 702 may drive additionalcameras and/or combinations of high- and low-resolution cameras thatacquire images at different frame rates.

Other arrangements are as possible, as evident to those of skill in theart. For instance, the image sensors on each arm may be equivalent toeach other, but the collection optics may be different. One arm mayinclude extra image magnification optics to ‘zoom in’ on a particularsubset of the image, providing a simultaneous wide-field and magnifiedview.

Illumination Configurations

The inventive visual inspection systems harness the manner in whichvarious particles interact with light to detect and identify particlesin fluid-bearing containers. The interaction of a particle with light isa complex function of a number of factors, including the particle'ssize, shape, refractive index, reflectivity and opacity. Proteinaceousparticles may primarily scatter light through refraction, while laminarglass particles may predominantly reflect light. Some particles, forexample collagen fibers, can modify intrinsic physical properties of thelight, such as a rotation of polarization. Tailoring the detector,particle, and light geometry to maximize contrast between variousparticle types can lead to highly accurate detection anddifferentiation.

FIGS. 8-12 show various illumination configurations that are tailored orcan be switched/actuated among different illumination modes for specifictypes of particle, container, and/or fluid. For example, the lightsources may illuminate the particles in such as way as to maximize theamount of light they reflect or refract towards the detector, whilekeeping the background dark to maximize the contrast between the imagesof the particles and the background. In addition, the sources may emitradiation at any suitable wavelength or range of wavelengths. Forexample, they may emit broadband white light (390-760 nm), a narrowbandbeam (e.g., at 632 nm), or even ultraviolet or X-ray radiation. Suitableranges include 10-3000 nm, 100-390 nm (ultraviolet), 390-760 nm(visible), 760-1400 nm (near infrared), and 1400-3000 nm (mid-wavelengthinfrared). X-ray emissions (<10 nm) are also possible. When taken as acomplete ensemble, the array of lighting options disclosed herein allowsinventive visual inspection systems to detect and identify the fullrange of particles that can potentially appear in drug products.

Because some particles scatter only very weakly, it is often beneficialto irradiate the sample with as much light as possible. The upper limitof the sample irradiance is primarily driven by the photosensitivity ofthe product under examination. A judicious choice of wavelength may alsobe necessary, particularly for biological products; the exact choicedepends on the product being illuminated. Monochromatic red lightcentered around 630 nm represents a ‘happy medium’ and is an easilyavailable wavelength in terms of affordable light sources.

LED arrays, such as the LDL2 series LED arrays from CCS Lighting, areeffective for illuminating particles seen in pharmaceutical products;however, collimated laser beams could also be used. In some cases,illumination optics may pattern or shape the illumination beam to becollimated inside the fluid volume (as opposed to outside thecontainer). For alternative light sources, if heating from the lightsource is a concern, light can be delivered to the inspection areathrough the use of optical waveguides or optical fibers 124 as shown inFIG. 8.

The illumination wavelength can be chosen based on the absorption and/orreflectivity of the fluid and/or particles being analyzed; this isespecially important light-sensitive pharmaceutical products. Red light(630 nm) offers a good balance between low absorption by the protein andlow absorption by water. Strobing the illumination in sync with thetimes-series data acquisition further protects the integrity oflight-sensitive pharmaceutical products by minimizing the products'exposure to incident light. Strobing has two further advantages: LEDsoperate more efficiently when run in this manner, and strobing reducesthe effect of motion blur, which left unattended compromises particlesize measurements as described below.

FIG. 8 shows an exemplary reconfigurable illumination system 120 thatincludes several light sources 122 a-122 f (collectively, light sources122), which may be LEDs, lasers, fluorescent or incandescent bulbs,flash lamps, or any other suitable light source or combination ofsuitable light sources. Light sources 122 may emit visible, infrared,and/or ultraviolet radiation. They may be narrowband or broadband asdesired, and can be filtered using appropriate optical filters orpolarizers. In FIG. 8, for example, a polarizer 126 polarizes lightemitted by the light source 122 f that backlights the container. Inaddition to the backlight 122 f, the illumination system 120 includesfour lights sources 122 a-122 d at corners of rectangular prism aroundthe container 10. Another light source 122 e illuminates the container10 from the bottom via an optical fiber 124 coupled to a collimator 126pointing at the bottom of the container 10. In some cases, the fiber 124and collimator 126 may be housed inside a hollow shaft 128 of thespindle used to rotate the vessel.

The multiple light sources 122 shown in FIG. 8 can be used to determinethe optical properties of a given particle for differentiation based onthe given particle's interaction with light. As understood by those ofskill in the art, different particles interact with light in varyingmanners. Common modes of interaction include scattering, reflecting,occluding, or rotating the polarization of the light, as shown in TABLE1, where “X” indicates that a particle of this type will show up using agiven lighting technique, as exemplified in FIGS. 9A-9D and FIG. 11(described below). An “M” indicates that particles of this type mightshow up using a given technique, but could still potentially bedetected/differentiated using post-processing image segmentation andfeature identification techniques.

TABLE 1 Light Interaction for Various Particle Types Particle TypeProtein Lamellae Opaque Cellulose Air Primary Interaction LightingPolarization Technique Scatter Reflect Occlude Change Scatter Rear AngleX X X X X Bottom X M Backlight X Polarizing M M X M

FIGS. 9A-9C illustrate different illumination patterns that can beimplemented with the illumination system 120 of FIG. 8 (some lightsources 122 are omitted for clarity) to differentiate particle typebased on light interaction. In FIG. 9A, light sources 122 a and 122 bprovide rear angled lighting, which is useful for showing proteins, aswell as most particle types that scatter light. In FIG. 9B, Light source122 e provides bottom light, which is useful for showing reflectiveparticles, such as glass lamellae, that reflect light towards the imager110 (horizontal arrow); particles that scatter but do not reflect light(e.g., proteins), may not show up on the sensor (diagonal arrow). InFIG. 9C, light source 122 f provides uniform backlight, which is usefulfor showing particles that occlude the light, such as metal, darkplastic, and fibers. Those of skill in the art will readily appreciatethat other light sources and/or illumination patterns and sequences arealso possible.

FIG. 9D shows how the lighting techniques of FIGS. 9A-9C can be appliedsequentially to capture time-series data of scattering, reflecting,and/or occluding particles. In this case, a system containing a uniformbacklight, rear-angled lights, a bottom light and a single cameraalternates the lighting each frame, such that only one particular lightsource 122 (or combination of light sources 122) is active at a time.For a single imager (not shown), only one set of lights is used peracquired frame of time-series data. Repeating this sequence provides avideo for each lighting configuration.

Acquiring a video sequence using the aforementioned lighting techniquessequentially provides a near simultaneous video for each light source122. At completion, this provides three interleaved videos, one for eachlighting technique. For each video, a particle in a given frame maycorrelate with the same particle in the other two videos using alternatelighting techniques (neglecting the small time difference betweenframes). Using the mutual information contained from the way a givenparticle interacts with the various lighting techniques, conclusions canbe made about the material composition of the particle.

This technique can be combined with other image feature extractioninformation in order to increase specificity. For instance, the videoscan be auto-segmented to determine the features in each frame. For eachlighting technique, information such as size, shape, brightness,smoothness, etc., can be automatically determined for each feature. Thiscan help to differentiate different particle types that have similarsignatures in terms of visibility on each of the different lightingtechniques.

FIGS. 10A-10C illustrate how to reduce glare caused by unwantedreflection/refraction of light from the light sources 122 off thecontainer 10. Illuminating the container 10 causes unwanted glare toappear in images captured by imagers 110 whose optical axes are alignedwith the propagation direction of light from the light sources 122 thatreflects off the container surface. Glare may obscure particles thatwould otherwise be detectable and saturate areas of the sensor.Positioning the imager 110 or the light sources 122 so that the imager'soptical axis is not coincident with or parallel to rays of light emit bythe light sources 122 that reflect of the container surface reduces oreliminates glare detected by the sensor. For example, placing the lightsource(s) 122 outside of an exclusion zone defined by revolving theimager about the longitudinal axis of the container 10 reduces theamount of unwanted reflected and/or refracted light captured by theimager. Alternatively, the zone 100 can be defined as a plane orthogonalto the central axis of the cylindrical container, with a thickness equalto the height of the containers' vertical walls. As understood in theart, containers with more complex shapes, such as concave sidewalls, mayhave different exclusion zones and different corrective optics.

Illuminating the container sidewalls obliquely from above or below thezone 1000, or from directly below the container base also reduces theglare detected by the imager 110. Illuminating the container 10 frombelow (e.g., with light source 122 e (FIG. 8)) also provides excellentcontrast between particles that reflect light (e.g., glass lamellae) andthose which scatter light (e.g., protein).

FIGS. 10D-10E illustrate an alternative illumination scheme for reducingor eliminating glare from the container 10, where one or more lightsources sources 122 are placed in the exclusionary zone described above(e.g., in the horizontal plane of the container 10).

FIGS. 10D-10E show a ray optics model of the propagation of rays outwardfrom the sensor of imager 110, through the imaging optics of the imager(as shown, including a telecentric lens), and back through the container10. A light source placed along any of the rays that back propagate fromthe sensor will refract or reflect light onto the sensor, therebypotentially obscuring the container 10 and its contents. Note however,that there are two regions 1001 located in the horizontal plane of thecontainer 10 and close to the outer wall of the container 10. As shownin FIG. 10E, if one or more light sources 122 are placed in the regions1001, glare from the light sources may be reduced or substantiallyelimination.

Note that, because a telecentric lens was used in the example shown,only light rays incident normal to the sensor need to be considered inthe ray optics model. However, a similar approach may be applied forother types of imaging optics, taking into account additional rays. Forexample, in some embodiments, one may back propagate a representativeset of rays from the sensor (e.g., including the principle rays of theimaging system) to identify regions that are free or substantially freeof back propagated rays. Illumination light sources can be placed in theidentified regions while avoiding glare.

FIG. 11 shows a setup for distinguishing elongated protein aggregatesfrom cellulose and/or fibers (natural or synthetic) with polarizedlight. An illumination system 120 emits light towards the container 10,which is sandwiched between crossed polarizers 900 that provide a blackimage in the absence of particles. Particles that modify (e.g., rotate)the polarization of the incident light appear white in the time-seriesdata detected by the imager 110.

If the particles of interest are known to fluoresce, fluorescenceimaging can be employed for particle identification, as shown in FIG.12. In this case, an illumination source 920 emits blue light that itexcites the particle of interest. A narrow-band (e.g., green) filter 922placed in front of the imager 110 ensures that only fluorescence fromthe excited particles will reach the detector. These illumination andfilter wavelengths can be selected to suit the specific wavelengths ofinterest.

Finally, it is possible to detect (and identify) particles, such assmall pieces of black, opaque material, that neither scatter (refract)nor reflect light. For such opaque particles, the sample should bebacklit directly from behind. The particles are then identifiable asdark features on a bright background. Images of opaque particles can beinverted, if desired, to form images that are scaled with same polarityas images of scattering and reflective particles (that is, so particlesappear as light spots on dark backgrounds instead of dark spots on lightbackgrounds).

Lamellae-Specific Visual Inspection Platforms

As understood by those of skill in the art, glass lamellae are thin,flexible pieces or flakes of glass formed by chemical reactionsinvolving the inner surfaces of glass containers. The inventive systemsand techniques can be used and/or tailored to detect, identify, andcount glass lamellae to minimize the likelihood of administering drugscontaining glass lamellae in order to prevent administration of drugscontaining (excessive quantities) of glass lamellae. Inventive systemsand techniques can also be used and/or tailored to study glass lamellaeformation, which depends on the makeup of a given formulation and differfrom proteins and other types of particulate matter in that they reflectand scatter light. Without being bound to any particular theory, itappears that certain conditions are more likely than others to promoteor hinder glass lamellae formation. For example, glass vialsmanufactured by tubing processes and/or under higher heat tend to lessresistant to lamellae formation than molded glass vials. Drug solutionsformulated at high pH (alkaline) and with certain buffers, such ascitrate and tartrate, are also associated with lamellae. The length oftime the drug product remains exposed to the inner surface of thecontainer and the drug product temperature also affect the chances thatglass lamellae will form. For more, see, e.g., the U.S. Food and DrugAdministration, Advisory to Drug Manufacturers: Formation of GlassLamellae in Certain Injectable Drugs (Mar. 25, 2011)(www.fda.gov/Drugs/DrugSafety/ucm248490.htm), which is incorporatedherein by reference in its entirety.

In order to create a system for differentiation based on this principle,the imager can be aligned with a vial in a typical fashion and orientedthe incident lighting through the bottom of the container (orthogonal tothe camera axis). This yields very little signal from particles thatscatter (e.g., proteins), and a large signal from particles that reflect(e.g., glass lamellae). In other words, as the lamellae float throughthe vessel, they appear to flash intermittently. This technique hasshown to be highly specific in differentiating lamellae particles fromprotein aggregates. Additionally, the signal obtained using this imagingtechnique is correlated with the concentration of lamellae within avial. As a result, this technique can not only be used fornon-destructive detection of lamellae in commercial products, but alsoused as a tool for determining which formulation compositions lead toincreased/decreased lamellae presence.

FIGS. 13A and 13B show maximum intensity projection (MIP) images ofglass lamellae (FIG. 13A) and protein (FIG. 13B) acquired with anillustrative visual inspection system. Conventional MIP images are usedin computerized tomography to visualize a three-dimensional space viewedalong one spatial axis, e.g., the z axis. A typical conventional MIPimage represents the maximum value of the data taken along an opticalray parallel to the visualization axis. In this case, however, the MIPimages shown in FIGS. 13A and 13B are visualizations of data thatrepresent the temporal evolution of a two-dimensional image—they areprojections along a temporal axis rather than a spatial axis.

To create the MIP images shown in FIGS. 13A and 13B, the processorselects the maximum value of at least some of the pixels in thetime-series data, where each pixel represents the amount of lightreflected (and/or transmitted) from a respective spatial location in thevessel. Plotting the resulting values yields a MIP image, such as thoseshown in FIGS. 13A and 13B, that represents the brightest historicalvalue of the pixels. The processor scores the MIP image by counting thenumber of pixels in the MIP image whose value exceeds a predeterminedthreshold. If the score exceeds a historical value representing thenumber of lamellae in a similar vessel, the processor determines thatthe vessel is statistically likely to contain glass lamellae. Theprocessor may also determine the severity of lamellae contamination byestimating the number, average size, and/or size distribution of theglass lamellae from the MIP image.

Inventive systems can also be used to distinguish glass lamellae fromother particles in the vessel, e.g., based on differences in the amountof light reflected by the particles as a function of time and/or ondifferences in the amount of light transmitted by the particles. Somenon-lamellae particles may reflect light from a light source thatilluminates the vessel from below (e.g., light source 122 e in FIG. 8)to the detector. Glass chunks, metal chunks, and foreign fibers, forinstance, could show up continuously using a bottom lightingconfiguration. These types of particles will consistently be detected asthey move through the container, as opposed to lamellae which areorientation dependent and only visible for a few frames each time theyalign themselves to reflect light towards the imager. Particle trackingcan be employed on bottom light time series images to track consistentlyvisible, yet moving, particulate matter. These tracks can then beeliminated from MIP calculations used for lamellae scoring, oralternatively be included in a mutual light information technique todetermine how a given particle interacts with other lightingorientations. For example, a metal particle that reflects light may betracked on the bottom lighting configuration. That same particleoccludes light when illuminated with a back light (e.g., light source122 f in FIG. 8). Using both of these metrics makes it possible todifferentiate the metal particle from a glass chunk, which reflectsbottom lighting but does not occlude rear lighting.

Particle Detection, Tracking, and Characterization

As described above, the visual inspection unit 100 shown in FIG. 1 canrecord a high quality, high-resolution monochromatic stream of images(time-series data) of bright particles imaged against a dark background.(Alternatively, the particles can be displayed as dark spots on a whitebackground.) Because drug product can contain a wide assortment ofradically differing particles, the time-series data can be analyzedusing a number of different approaches to differentiate features on animage from the background. Often, the appearance of a particle on asingle image (frame of time-series data) is not sufficient to make trulyaccurate quantitative estimates for critical objectives (e.g.,count/size). For instance, what appears to be a single particle in oneframe of time-series data may actually be two or more particlescolliding with each other or passing by each other, which may result inaccurate particle counts and/or estimations of particle size.

Temporal correlation of image features between frames in a videosequence improves the precision of particle counting and sizemeasurements. The process of linking image features in consecutiveframes together to form a time dependent trajectory for each particle isknown as particle tracking, registration, or assignment. Particletracking techniques exist for other applications (notably in theexperimental study of fluid mechanics). However, these applicationstypically employ well-defined spherical tracer particles. Applying theprinciple to drug products and other fluids requires a significantlymore complex solution. In addition, for some species of particles,temporal (tracking) analysis is not always practical. In such cases astatistical approach can be employed as an alternative to yieldcharacteristic measurements.

FIG. 14 provides an overview of the high-level particle detection andidentification 1300, which starts with acquisition 1310 of time-seriesdata. The time-series data (and/or reversed time-series data) ispre-processed 1320, and the pre-processed, reversed time-series data isused for two-dimensional particle identification and measurement 1330,which may include statistical analysis 1340 and/or particle tracking1350 of the reversed time-series data. As explained above, reversedtime-series data is time-series data whose frames have been re-orderedin reverse chronological order. Particle report generation 1360 occursupon completion of particle identification and measurement 1330.

Time-Series Data Pre-Processing

Pre-processing 1320 includes static feature removal (backgroundsubtraction) 1321, image noise suppression/filtering 1322, and intensitythresholding 1323. Static feature removal 1321 exploits the fact thatspinning the container energizes the fluid and the particles containedwithin. Their dynamic motion allows them to be distinguished from otherimaging features. Since image capture commences after the container hasstopped spinning, the assumption is that everything that is moving is apotential particle. Static features are subsequently irrelevant and canbe removed from the image to improve clarity.

In one embodiment, a minimum intensity projection establishes anapproximate template for features in the image that are static. Thisincludes, for instance, scratches, dirt and defects that may be presenton the container wall. This ‘static feature image’ can then subsequentlybe subtracted from the entire video sequence to generate a new videosequence that contains only moving features against a black background.For example, FIGS. 15A and 15B show a single frame of time-series databefore and after static feature removal. Glare, scratches, and otherstatic features obscure portions of the container in FIG. 15A.Background subtraction removes many of the static features, leaving animage (FIG. 15B) with more clearly visible moving particles.

A caveat of this approach is that most glass defects such as surfacescratches scatter a relatively significant amount of light, appearingbright white in the captured images, as detector pixels are saturated.Subtraction of these features may result in ‘dead’ regions in the image.As particles move behind or in front of these illuminated defects, theymay be partially occluded or even disappear entirely. To resolve thisproblem, the ‘static feature image’ can be retained, analyzed, and usedto correlate defect positions to particle positions to minimize theinfluence of surface defects on particle size and count data. (As a sidenote, application of a cleaning protocol is advised before operating thesystem to ensure surface defects have been removed as much as possible.)The data can also be filtered 1322, e.g., to remove high-frequencyand/or low-frequency noise. For example, applying a spatial bandpassfilter to the (reversed) time-series data removes and/or suppresses datathat varies above a first spatial frequency or second spatial frequency.

Once the background features have been removed, the time-series data isthresholded 1323 by rounding the intensity value of each pixel in theimage to one of a predetermined number of values. Consider the grayscaleimages shown in FIGS. 16A and 16C, which are scaled according to theeight-bit scale shown at left (other possible scales include 16-bit and32-bit). Each pixel has an intensity value from zero to 255, where zerorepresents no detected light and 255 represents the highest amount oflight detected. Rounding those intensity values of 127 or under to zeroand those intensity values of 128 and up to 255 yields theblack-and-white images shown in FIGS. 16B and 16D. Those of skill in theart will readily appreciate that other thresholds (and multiplethresholds) are also possible.

Particle Detection

Effective particle detection in an image relies on a variety of imageprocessing and segmentation techniques. Segmentation refers to thecomputational process by which features of interest in an image aresimplified into discrete, manageable objects. Segmentation methods forextracting features from an image are widely used, for example, in themedical imaging field, and these techniques have been employed forparticle identification. In short, the images acquired from the cameraare pre-processed using thresholding, background (static feature)subtraction, filtering (e.g., bandpass filtering), and/or othertechniques to maximize the contrast. At the completion, the processor130 segments the image, then selects certain areas of an image asrepresenting particles and categorizes those areas accordingly. Suitablesegmentation approaches include, but are not limited toconfidence-connected, watershed, level-set, graph partitioning,compression-based, clustering, region-growing, multi-scale, edgedetection, and histogram-based approaches. After the images areacquired, segmentation can yield additional information to correlate agiven feature on an acquired image with a particle type. For instance,information about the given segmented feature such as area, perimeter,intensity, sharpness, and other characteristics can then be used todetermine the type of particle.

Particle Tracking and Time Reversal

Critically, no previously available particle identification toolsconsider in full detail the temporal behavior of the particles as theymove around the vial. The counting and sizing of particles can beinaccurate if only measuring from a single “snapshot.” However,time-series data provide a more complete picture of particle behaviorthat can be resolved using using particle tracking 1340, which enablesthe creation of time-dependent spreadsheets for each individualparticle, enabling a far more robust and accurate measurement of itsfundamental properties. Particle tracking is a technique usedextensively in video microscopy, as well as in fluid dynamicsengineering (where it is commonly referred to as particle trackingvelocimetry, or PTV).

Although PTV is known, the majority of particle tracking solutionsassume that movement of particles between successive video frames isslight, and smaller than the typical separation distance betweenparticles in a given image. In such cases, it is sufficient to linkparticle positions by identifying closest matching neighbors. In manyapplications, however, this is not an appropriate model. Due to the spinspeed (e.g., about 300 rpm, 1600 rpm, and/or 1800 rpm) and potentiallyhigh particle concentrations, particles can be expected to move farfurther between successive frames than the typical inter-particleseparation distance. This can be resolved by employing a form ofpredictive tracking, which involves searching for a particle in a regionpredicted by the particle's prior movement. Predictive tracking includesthe evaluation of physical equations to mathematically predict theapproximate future position of the particle in the subsequent frame, asshown in FIG. 17. For improved performance, this phase of predictivetracking can be coupled with knowledge of the local fluid behavior (ifknown), e.g., as described with respect to FIG. 21C.

Forming an accurate prediction for a given trajectory may require someprior data points on which to base the trajectory. This presents aconundrum—at the start of the image sequence, when the particles aremoving fastest, there may be little to no prior data on which to baseposition predictions. Over time, however, wall drag in the containercauses the rotating fluid to slow down and ultimately stop. Recordingtime-series data for long enough yields frames in which the particlesslow down considerably and even stop.

Reversing the timeline of the video 1331, so that the particlesinitially appear to be static, and slowly speeding up as the videoprogresses provides “prior” data points for determining the trajectory.At the start of the video, where the particles are now barely moving,the nearest-match principle can be used to build up the initial phase ofeach trajectory. At an appropriate time, the system can then switch tothe predictive mode. Reversing the timeline of the acquired data in thismanner dramatically improves performance.

FIG. 17 shows an overview of predictive tracking with time reversal. Thegoal of particle tracking is to track link the position of a particlea_(i) in frame i to its position a_(i+1) in frame i+1, as shown in FIG.17(a). This is straightforward if the movement of particle a betweenframes is smaller than the distance d to its nearest neighbor, particleb. If the particle's direction of movement is unknown or random, thesimplest methodology is to have a search zone—typically a circle ofradius r_(s), where r_(s) is chosen so as to be longer than the expectedrange of particle movement, but smaller than the typical inter-particleseparation distance d, as shown in FIG. 17(b). After reversing the movietimeline, as in FIG. 17(c), the particles appear to begin to moveslowly. After a while, however, the particles appear to speed up, andthe nearest-match search method may start to fail. The first few framesof reversed time-series data partially establish the trajectory,yielding some knowledge of the particle's velocity and acceleration.This information can be input into appropriate equations to predict theparticle's approximate location in frame i+1, as in FIG. 17(d). Thispredictive tracking method is considerably more effective than simplenearest match tracking, especially in dense and/or fast moving samples.

Center-of-Mass Detection

FIGS. 18A and 18B illustrate center-of-mass detection for particles in(reversed) time-series data after thresholding. First, the processor 130transforms a grayscale image (FIG. 18A) into a thresholded image (FIG.18B). Each particle appears as a two-dimensional projection whose shapeand size depend on the shape, size, and orientation of the particle whenthe frame was recorded. Next, the processor computes the geometriccenter, or centroid, of each two-dimensional projection (e.g., asindicated by the coordinates x_(i) and y_(i)) using any suitable method(e.g., the plumb line method, by geometric decomposition, etc.). Theprocessor 130 can compare the location of the centroid of a particularparticle on a frame-by-frame basis to determine the particle'strajectory.

Particle Occlusion

Each of the visual inspection systems disclosed herein project athree-dimensional volume—a container and its contents—onto thetwo-dimensional surface of the image sensor. For a given two-dimensionalsensor, it is possible for particles in the three-dimensional volume toappear to cross paths. When this happens, one particle may partially orcompletely occlude another, as shown in FIG. 19. In FIG. 19(1), a newparticle is identified in the image sequence; tracking the particlethrough the image sequence yields a series of sequential steps as shownin FIG. 19(2). Employing a search zone to look for potential matches inconsecutive frames as shown in FIG. 19(3). Occasionally more than onecandidate particle will occupy the search zone, as shown in FIG. 19(4),in which case the system selects the best match. As readily appreciatedby those of skill in the art, the best match can be decided using anyone of combination of different approaches. For instance, datarepresenting a candidate particle in one frame can be compared to and/orcorrelated with data representing a particle in a preceding frame.Comparing and/or correlating parameters including, but not limited tosize, shape, brightness, and/or change in appearance leads to a matchfor the candidate particle. Illustrative visual inspection systems cancope with collisions, occlusions, and temporary particle disappearances,such the occlusion shown in FIG. 19(5). When the particle is recovered,as in FIG. 19(6), the track can be reconstructed. Illustrative systemscan also resolve conflicts caused when two tracks (and their searchzones) collide, ensuring that the correct trajectories are formed, as inFIG. 19(7).

FIG. 20 illustrates another case of particle occlusion in atwo-dimensional image: (a) is a typical image of particles insuspension. FIGS. 20(b)-(e) show close-ups of the boxed region in FIG.20(a), with two particles approaching one another from oppositedirections. The next frames in the (reversed) time-series data show thatocclusion causes two particles to appear to be a single, artificiallylarge particle. If the occlusion is partial (FIG. 20(c)), this can leadto the appearance of single, artificially large particle. If theocclusion is complete (FIG. 20(d)), then the smaller particle may belost from the field of view completely and the particle count maydecrease by one. This may be of critical importance when inspecting drugproducts because the artificially increased size measurement may besufficient to exceed regulatory thresholds, when in fact the productunder scrutiny contains only acceptable, sub-visible particles. By FIG.20(e), the particles have moved beyond one another and independenttracking can continue. By analyzing the particle trajectories and thesubsequent time-dependent size profiles, the visual inspection systemcan automatically correct for errors due to occlusion, leading to alower rate of false rejects.

Accounting for Lost Particles

As discussed, particles can disappear from a portion of a given videosequence for a number of reasons. They may traverse a ‘blind spot’and/or a ‘dead’ region due to the static feature removal as discussedabove. Finally, some types of particles may exhibit optical behaviorwhere they appear and disappear (sparkle) with respect to the imagingoptics. In such cases, the processor can predict the movement of these‘lost particles’ as follows. Should the particle re-appear at anexpected location within a certain timeframe, the processor can link thetrajectories and interpolate virtual particle data for the interimframes. Note that from a regulatory standpoint it is important to beclear that virtual particle data is appropriately tagged so that it canbe distinguished from true measured particle data.

FIGS. 21A-21C illustrate one technique for tracking and recovering lostparticles, i.e., particles that temporarily disappear from the field ofview over the course of a video sequence. Disappearance may be due toocclusion behind another (larger) particle, occlusion behind a surfacedefect, transition through a known blind spot or simply a property ofthe particle's optical geometry (for example, some types of particlesmay only be visible at specific orientations). Finding or recoveringparticles that disappear from the field of view improves the precisionwith which particles can be detected and identified.

FIG. 21A illustrates predictive tracking to find a particle that isoccluded by a defect on the container surface. The surface defectscatters a large amount of light, saturating the corresponding region ofthe image. After static feature removal is employed, this results in a‘dead zone’ in the image. Any particles that traverse this zonedisappear temporarily. The processor 130 can recover ‘lost’ particles bycreating virtual particles for a finite number of steps. If the particlere-appears and is detected, the tracks are united.

More specifically, the processor 130 uses predictive tracking todetermine the particle's velocity prior to its disappearance. It canalso use predictive tracking and the particle's velocity to extrapolatean expected particle position. If the particle appears again in anexpected position, the virtual positions can be linked to form acomplete trajectory. If the particle does not reappear within apre-defined time window, it can be signaled as being permanently lost,and is no longer tracked.

FIG. 21B shows how to track a particle that undergoes a significantacceleration or change of direction while it is out of sight. Ratherpredicting the particle trajectory, the processor 130 retrospectivelylinks fragmented trajectories using the nature of the local behavior ofthe fluid. In this case the processor 130 united the trajectories byconsidering the laminar flow characteristics of the fluid at this speedand scale.

FIG. 21C illustrates how particles disappear and re-appear as theytraverse known blind spots. In this example, the particle traverses aknown blind spot at the extreme edge of the container. Programming theprocessor 130 with information about the position of the blind spot withrespect to the container image enables the processor 130 to reconstructthe trajectory.

Particle Shape Irregularity

Some particles are not spherical or small enough to be consideredpoint-like, as assumed by most particle tracking techniques. In fact,many particles are irregularly shaped and may tumble and rotate relativeto the camera as they move through the fluid, as shown in FIGS. 22A-22C.In some cases, an irregularly shaped particle may appear as two separateparticles, each with its own trajectory, as shown in FIG. 22B. Suchunpredictable movement of the measured center of mass of thetwo-dimensional object may obscure the true movement of the particle.This behavior seriously complicates the process of predictive tracking.The visual inspection system described herein may contain functionalityto cope with the apparently perturbed motion of an irregularly shapedparticle, e.g., by calculating a mean trajectory as shown in FIGS. 22Aand 22C for the irregularly shaped particle.

Container/Product-Specific Fluid Dynamics

The motion of particles in the container post-spin is a result of thecombination of the motion of the fluid with the effect of gravity. Themotion of the fluid is a function of the fluid's viscosity, the fillvolume, the container shape and size, and the initial spin speed.Particle tracking performance can be significantly improved byincorporating knowledge of the physical constraints of the fluidicsystem into trajectory-building.

The fluid dynamics of liquids spun in conventional containers can besurprisingly complex under certain circumstances. Incorporating fluiddynamics knowledge (as it pertains to containers typically used in thedrug industry) into trajectory-building constitutes a significant areaof novelty and development over the prior art.

FIG. 23 shows some examples of fluid behavior in typical containers,with results from a computational model compared against real-worldparticle trajectories generated by the visual inspection platform.Studies have uncovered unexpected subtleties: as an example, in FIG.23(d) we can see particle movement along a narrow vertical column in thecenter of the vial, which is due to the relaxation of the vortex createdduring the spin phase (FIG. 23(a)). As the fluid in this central columnmoves vertically upwards, it can sweep upwards heavy particles that onemay normally expect to sink. This could, for example, cause confusionbetween identifying bubbles, which one would expect to rise, and foreignparticles which are rising due to container-specific fluid motion.

Illustrative visual inspection systems can leverage prior knowledge ofthe expected fluid dynamics of drug products to yield considerable moreaccurate results than would otherwise be possible. Combining a physicalmodel, such as the one illustrated in FIG. 23, with particle tracking inthis fashion represents a significant improvement over existingtechnology.

Error Correction

While the visual inspection systems disclosed herein are robust undermost experimental conditions, the complexity of the challenge oftracking large numbers of particles moving in a small three-dimensionalvolume means there is always the risk of some errors being introduced,chiefly in the form of incorrect trajectories being formed betweensuccessive frames when particles ‘collide’. This phenomenon isillustrated in FIG. 24A.

An understanding of the physical constraints of the visual inspectionsystem can be employed to advantage. Broadly speaking, the predominantmovement of the fluid locally around each particle is laminar (ratherthan turbulent or random). What this essentially means is that, with asufficiently fast camera, natural particle trajectories in this systemshould be smoothly varying, with no sudden, sharp changes in direction,particularly as particles traverse the center of the container in theimage. Once initial trajectory linking is complete, the system canretrospectively analyze the trajectories for such errors. If they aredetected, the system can compare nearby trajectories to establishwhether a more physically consistent solution can be found. This isshown in FIG. 24B.

Accurate Particle Counting

A particle count can be deduced by counting the number of particles in asnapshot image taken at a single point in time (e.g., as shown in FIG.24A) after particle detection, where each particle is labeled with acount number. This approach is straightforward, but has a tendency tosystematically undercount the number of particles in the volume for avariety of reasons. For instance, one or more particles may be occludedby another particle or surface defect. Particles may be in known (orunknown) blind spots. In addition, extremely small or faint particlesmay intermittently appear and disappear from view as they move acrossmeasurement thresholds.

One advantage of the particle tracking discussed herein is that it canaccount for all of these problems. As a result, for robust particletracking, particle counting can be improved by counting the number ofindividual particle tracks (as in FIG. 24B), rather than the number ofparticles in a single image or a statistical analysis of several images.Counting the number of particle trajectories rather than the number ofparticles in a single frame (or ensemble of frames) represents asignificant improvement over conventional particle tracking techniques.The size of the improvement varies with the number and size(s) of theparticles present. Roughly speaking, as the number of particlesincreases, the chance of occlusion increases and so the improvement dueto the temporal capabilities of the inventive particle tracking increaseproportionally.

Accurate Particle Sizing

Conventional particle measurement systems measure particle size fromstatic images. Most typically this is done by measuring the length ofthe particle's longest apparent axis, or Feret diameter, as shown inFIG. 25, according to regulatory and/or industry standards, which maydefine the particle size as the longest single dimension of theparticle. Under this definition, a 1 mm hair is classed the same as aspherical particle with a 1 mm diameter. With this in mind, from atwo-dimensional image, the maximum Feret diameter is a reasonablemeasurement to use. Measurement of particle size from static imageshowever, suffers several critical problems.

First, in a two-dimensional projection of a three-dimensional volume, itis easily possible for multiple particles to overlap, creating whatappears to be a single, much larger particle. In an industry whereregulators set very strict upper limits on allowable particle size, thisis a critical problem, particularly for manufacturing applications,where it may lead to false rejects, particularly for densely populatedsamples.

Second, irregularly-shaped particles may tumble unpredictably (relativeto the camera) as they flow around the container. With a singletwo-dimensional snapshot, it may be impossible to guarantee that a givenparticle's longest dimension is orthogonal to the camera's viewing axis.The system may therefore systematically undersize particles, which couldhave dire consequences in a heavily regulated industry. Examining thetime-dependent maximum Feret diameter of the particle as it flows aroundthe container through particle tracking provides a much more accuratemeasure of the particle's largest dimension.

Third, as particles move around a cylindrical container, they generallyalign their long axis with the direction of the surrounding fluid flow,as shown in FIGS. 25A and 25B. In general, for a cylindrical containerthis means that elongated particles may appear larger in the center ofthe image than at the extreme lateral edges. Usually, the imager detectsthe maximum apparent particle size (Feret diameter) when the particle istravelling orthogonally with respect to the optical axis of the imagesensor. If a single particle is tracked as it flows around thecontainer, its correct maximum elongation can be accuratelymeasured—something that is difficult for a static measurement procedureto achieve.

Finally, despite efforts to minimize the effect of motion blur bystrobing the illumination (as discussed above), it may still be possiblefor some degree of motion blur to occur at the start of the imagecapture sequence, when the fluid and particles are moving fastest. Byusing a time-dependent analysis of particle size, artifacts in the datadue to motion blur (which tends to increase measured particle size) canbe identified and suppressed.

FIGS. 25C-25E illustrate the use of time-series data to track particletrajectories for more precise particle size measurements. FIG. 25C showsthe typical track of a 100-micron polymer microsphere moving around avial post-spin. Particles move fastest relative to the camera as theyappear to cross the center of the container, when their velocity isorthogonal to the viewing direction, as shown in FIG. 25D. For example,if the initial spin speed is 300 rpm, and the radial position of theparticle r_(p) is 5 mm, then the particle velocity v_(p) is about 9.4m/s. At this speed, a camera exposure time of only 10 μs doubles theapparent particle size due to motion blur. FIG. 25E shows how badlymotion blur can affect images: at left, the particles are moving fast(about 300 rpm) and stretched out; at right, the same particles are at astandstill and appear more circular.

FIG. 25F is a graph of the time-dependent Feret diameter for theparticle shown in FIG. 25C. Due to lensing effects of the cylindricalcontainer, the particle's apparent size is reduced near the edge of thecontainer (right axis tick D). The best estimate of the maximum particlesize occurs when the particle traverses the center of the container, atmodest speed (right axis tick B). If the speed is too high (whichtypically occurs during the first few seconds after the container spin)then motion blur exaggerates the particle size (right axis tick A).Eventually, due to fluid drag the particle will stop moving altogether(right axis tick C). In this case, the mid-range peak values (right axistick B) is the most accurate reading of the maximum particle size.

Particle Characterization

FIG. 26A shows successive frames of time-series data with both theparticles and their trajectories. The roughly planar tracks representtrajectories of 100-micron polymer microspheres that mimic proteinaggregates. These particles, which are almost neutrally buoyant movewith the fluid and do not noticeably sink or rise. The verticallydescending tracks represent the trajectories of 100-micron glass beads,which rotated with the fluid initially but sank as the sequenceprogressed. Rising tracks represent the trajectories of air bubbles andparticles with positive buoyancy.

Particle tracking enables measurement of a number of time-dependentproperties that can give important clues as to the nature of theparticles under examination. For example, air bubbles, which cangenerally be considered benign from a regulatory standpoint, can confusecurrent optically-based inspection machines, leading to false positivesand unnecessary rejects. In this case, the time-dependent motion of theparticle (air bubbles tend to rise vertically as the fluid begins toslow down) leads to a very obvious characteristic that can easily beidentified from the trajectory produced by the particle tracking.Similarly, neutrally buoyant particles may not rise or fall much,whereas dense particle sink to the bottom of the container. Lighterparticles may be swept up in a vortex formed by the spinning fluid, andheavy particles may have straight-line trajectories.

More broadly, the particle tracking process produces a time-dependentspreadsheet, such as the one shown in FIG. 26B, that contains details ofall relevant parameters, including position, velocity of movement,direction of movement, acceleration, size (e.g., two-dimensional area),size (maximum Feret diameter), elongation, sphericity, contrast, andbrightness. These parameters provide a signature that can be used toclassify a particle as a particular species. This approach, which isachievable via a particle tracking solution, works well for mostparticles of interest. The ability to categorize particles, on aparticle-by-particle basis, based on such an array of time-dependentmeasurements is a particular benefit of the present invention.

Video Compression

Visualizing very small particles in a comparatively large containerbenefits from the use of very high resolution image sensors. The rate ofimage capture also needs to be maximized to ensure accurate trajectorybuilding. The combination of these requirements results in extremelylarge video files, e.g., 1 GB, 2 GB, 5 GB, 10 GB, or larger. For someapplications, it may be necessary to archive original video in additionto the analysis data. For even moderately sized sample sets, the largefile sizes involved could potentially make data storage costsprohibitive.

Video compression of the (reversed) time-series data can be used toreduce the sizes of (reversed) time-series data files. Protectingparticle data integrity may require the use of lossless videocompression. Studies suggest that more commonly used (and moreefficient) lossy compression techniques (e.g., MPEG) can criticallydistort and perturb the image, introducing a number of unwanted visualartifacts.

While lossless compression is, in general, comparatively inefficientcompared to lossy compression, there are a number of steps that canimprove its effectiveness. Most frames of the time-series data show ahandful of small, bright object set against a dark background. The darkbackground contains no useful information. It is not truly black—ratherit is made of very faint random noise. Replacing this background with apurely black background greatly simplifies the image, and makes it muchmore efficient for standard lossless compression techniques (e.g. zip,Huffyuv) to operate.

This process has been reported elsewhere in the literature. What isnovel here, however, is the specific decision of what actuallyconstitutes the background in a given frame. Other compression processesset a threshold intensity level and assume that all pixels in the imagebelow this level are part of the background. This is a broadly effectivestrategy but can result in a slight reduction in the size of retainedparticles, and can completely remove very faint particles whosebrightness is of the same order as the upper limit of the intrinsicrandom background ‘noise’.

Although these conventional techniques work with (reversed) time-seriesdata, the compression used in illustrative embodiments employs a uniquephase that analyses the background for faint particles prior to theemployment of destructive thresholding. This ensures the best balance ofretaining data integrity while maximizing reductions on data storagerequirements.

Fill Volume/Meniscus Detection

Automated embodiments of the visual inspection platform detect the fillvolume of the sample accurately, which is important in researchapplications, where there is no guarantee that the fill volume will beconsistent across a particular run. This is especially useful whendealing with very large data files, such as those generated byhigh-resolution image sensors, causing pressure on data transfer andstorage. For this reason, it can be desirable to limit the recordedimage to cover no more than the fluid volume, since any furtherinformation is irrelevant.

Illustrative systems may employ, for example, automated edge detectionor feature recognition algorithms to detect the boundaries of thecontainer in the image as shown in FIGS. 27-29 and described below.Because both the meniscus and the vial base are singular, uniquefeatures, a number of possible lighting configurations and/or imageprocessing techniques can be employed to accurately identify theirposition in the image. Measuring the fill volume and determining theregion of the image occupied by the fluid yields the region of interest.Specifically, from FIG. 8, configurations using light sources 122 f(backlight), 122 e (bottom light) and a combination of 122 a and 122 b(rear-angled lighting) can all be used to detect the fill volume asdescribed below.

FIGS. 27A-27F illustrate automatic detection of a region of interestwithin a container using the rear-angled lighting 122 a and 122 b inFIG. 8. FIG. 27A shows a static image of the container where the base ofthe vessel and the meniscus are clearly visible as distinct, brightobjects. As an example, a processor can employ edge detection toidentify the vertical walls of the container and width of the region ofinterest, w, as shown in FIG. 27B. For detection of the meniscus andvial base, whose appearance can be less predictable, the process can,for example, employ intensity thresholding and segmentation to provide asimplified image of the region of interest (shown in FIG. 27C). At thisphase, the processer can automatically identify containers that may notbe suitable for particle analysis, e.g., containers whose surfaces arescratched and/or covered in dirt. The effectiveness of the system can becompromised by excessive turbidity, container surface defects, orexcessively high particle concentration (whereby individual particlescan no longer be discretized in the image). If the processor determinesthat the container is satisfactory, the objects that correspond to themeniscus and the vial base can then be isolated and simplified as shownin FIG. 27D. The processor defines the vertical height h of the regionof interest as the distance between the lower edge of the meniscus andthe upper edge of the vial base as shown in FIG. 27E. Finally, theprocessor may crop the original image stream using the width and heightof the region of interest dimensions so that only the area of the imageoccupied by the visible fluid is recorded as shown in FIG. 27F.

FIGS. 28A-28C illustrate a similar meniscus detection process carriedout with data acquired using a backlit configuration (e.g., light source122 f in FIG. 8). FIG. 28A shows a frame of time-series datarepresenting a typical container imaged with a backlight. The meniscus,walls and base are clearly distinguishable, and can be automaticallyidentified using edge detection as in FIG. 28B. However, defects such aslarge scratches can potentially compromise the accurate detection of themeniscus position whether using a backlight (FIG. 28B) or therear-angled lights (e.g., as in FIG. 29C, described below). In oneimplementation, we use intensity thresholding of the image to identifythe meniscus and vial base. Since these are relatively large objects,and due to their shape scatter a relatively large amount of lighttowards the detector, they can be clearly identified, distinct from anyother features that may be present.

FIGS. 29A-29D illustrate detection of a meniscus in a cylindrical vesselwith a roughly planar bottom. Automated fill volume detection startswith thresholding (FIG. 29A) to detect the meniscus, which then sets theregion of interest and is also a measure of fill volume. Next, in FIG.29B, oblique lighting highlights surface defects such as scratches(shown), dust, fingerprints, glass defects or condensation can make edgedetection difficult. Lighting the vial from below (e.g., using lightsource 122 e as in FIG. 8), as in FIG. 29C, illuminates the meniscus ina manner which is (relatively) insensitive to surface defects—here, themeniscus is visible even though the surface is heavily scratched.Lighting from below also makes it possible to differentiate betweenempty vials and full vials, as shown in FIG. 29D, and to accuratelydetect the meniscus height at all fill levels between those extremes.Illuminating a vial from below increases the effectiveness of themeniscus detection, since it mitigates errors due to scratches and othersurface defects (FIG. 27C). Setting the light source 122 e to illuminatethe vessel at a slight angle further decreases the sensitivity tosurface defects. For syringes, which may be difficult to illuminate frombelow due to the absence of a transparent container base, a similareffect can be achieved by illuminating obliquely at a narrow angle.

Inspection techniques similar to the meniscus detection described abovecan also be employed to screen for features that would undermine anysubsequent attempts to identify and analyze particles suspended in thefluid. This may include the identification of excessively turbidliquids, critically damaged containers (including excessive scratchingor surface debris) and fluids in which the particle concentration is sohigh particles can no longer be discretized.

Processors and Memory

Those of skill in the art will readily appreciate that the processorsdisclosed herein may comprise any suitable device that providesprocessing, storage, and input/output devices executing applicationprograms and the like. Exemplary processors may be implemented inintegrated circuits, field-programmable gate arrays, and/or any othersuitable architecture. Illustrative processors also be linked throughcommunications networks to other computing devices, including otherprocessors and/or server computer(s). The communications network can bepart of a remote access network, a global network (e.g., the Internet),a worldwide collection of computers, Local area or Wide area networks,and gateways that currently use respective protocols (TCP/IP, Bluetooth,etc.) to communicate with one another. Other electronic device/computernetwork architectures are also suitable.

FIG. 30 is a diagram of the internal structure of an illustrativeprocessor 50. The processor 50 contains system bus 79, where a bus is aset of hardware lines used for data transfer among the components of acomputer or processing system. Bus 79 is essentially a shared conduitthat connects different elements of a computer system (e.g., processor,disk storage, memory, input/output ports, network ports, etc.) thatenables the transfer of information between the elements. Attached tosystem bus 79 is I/O device interface 82 for connecting various inputand output devices (e.g., keyboard, mouse, displays, printers, speakers,etc.) to the processor 50. Network interface 86 allows the computer toconnect to various other devices attached to a network. Memory 90provides volatile and/or nonvolatile storage for computer softwareinstructions 92 and data 94 used to implement an embodiment ofillustrative visual inspection systems and techniques. Disk storage 95provides (additional) non-volatile storage for computer softwareinstructions 92 and data 94 used to implement an embodiment ofillustrative visual inspection. Central processor unit 84 is alsoattached to system bus 79 and provides for the execution of computerinstructions.

In one embodiment, the processor routines 92 and data 94 are a computerprogram product (generally referenced 92), including a computer readablemedium (e.g., a removable storage medium such as one or more DVD-ROM's,CD-ROM's, diskettes, tapes, etc.) that provides at least a portion ofthe software instructions for illustrative visual inspection systems.Computer program product 92 can be installed by any suitable softwareinstallation procedure, as is well known in the art. In anotherembodiment, at least a portion of the software instructions may also bedownloaded over a cable, communication and/or wireless connection. Inother embodiments, exemplary programs are a computer program propagatedsignal product 107 embodied on a propagated signal on a propagationmedium (e.g., a radio wave, an infrared wave, a laser wave, a soundwave, or an electrical wave propagated over a global network such as theInternet, or other network(s)). Such carrier medium or signals provideat least a portion of the software instructions for the illustrativeroutines/program 92.

In alternate embodiments, the propagated signal is an analog carrierwave or digital signal carried on the propagated medium. For example,the propagated signal may be a digitized signal propagated over a globalnetwork (e.g., the Internet), a telecommunications network, or othernetwork. In one embodiment, the propagated signal is a signal that istransmitted over the propagation medium over a period of time, such asthe instructions for a software application sent in packets over anetwork over a period of milliseconds, seconds, minutes, or longer. Inanother embodiment, the computer readable medium of computer programproduct 92 is a propagation medium that the processor 50 may receive andread, such as by receiving the propagation medium and identifying apropagated signal embodied in the propagation medium, as described abovefor computer program propagated signal product.

Generally speaking, the term “carrier medium” or transient carrierencompasses the foregoing transient signals, propagated signals,propagated medium, storage medium and the like.

Sensor Cooling

In the above-described embodiments, electronic sensors are used tocapture images of particles. Electronic sensors such as CCDs are subjectto several types of random noise which serve to compromise the integrityof the measured signal, especially at low signal strengths. In someembodiments the sensors may be cooled to reduce noise. The cooling maybe accomplished using any suitable technique, including, e.g., the useof thermoelectric coolers, heat exchangers (e.g., cryocoolers), liquidnitrogen cooling, and combinations thereof.

In various embodiments, the noise reduction has an advantage in particledetection, especially relating to the detection of protein aggregates.In typical applications, protein aggregates can be relatively large(e.g., up to several hundreds of microns in diameter) however thephysical structure of these aggregate particles is often very loose,with low density (a large proportion of the particle may be porous andfilled with the surrounding medium) and of low refractive index contrastto the surrounding medium. Due to these physical properties, proteinaggregates can scatter relatively small amounts of light compared toother particles, such as glass fragments or fibers.

Much of the noise affecting contemporary electronic image sensors isthermal in nature. This noise primarily affects the lower end of thedynamic range of the sensor. For example, in some embodiments, the lowerX % (e.g., 10%) of the dynamic range is occupied by noise and must beremoved during the image thresholding process (e.g., as describedabove). The threshold value for particle detection must be, at minimum,higher than this value of ˜X %, thereby removing low intensity data fromthe signal. This may prevent the accurate detection of faint particlessuch as protein aggregates. By reducing the noise, e.g., by cooling thesensor, a lower threshold value may be used, allowing for improveddetection of low intensity signals.

FIGS. 31A-31D illustrates the thresholding issue described above. FIG.31A shows a cropped segment from a typical image sequence acquired usingthe devices and techniques described herein. As shown, the images are8-bit grayscale images, that is, each pixel can have an intensity valueranging linearly from 0 (black) to 255 (white). The image contains twoparticles, one relatively bright and one very faint. FIG. 31B shows anintensity histogram showing the intensity values of the‘background’—corresponding to the box in the image that does not containany particles.

The sensor exhibits a Gaussian background noise curve at the low end ofthe intensity histogram, due at least in part to thermal effects. Thewidth of this curve determines the threshold value for particledetection. In short, particles need to be significantly brighter thanthe background noise to survive thresholding.

FIG. 31C shows an intensity histogram for the bright particle. Theparticle image has a significant number of pixels to the right of thethreshold value in the histogram and so will be easily detectable afterthresholding.

In contrast, as shown in FIG. 31D, the fainter particle has a relativelysmall number of pixels above the threshold value—it would likely bewiped out during the thresholding/segmentation process. However, ifcooling or other techniques were applied to reduce the noise floor,thereby shifting the threshold value to the left, it is possible thatthe fainter particle could be detected.

Light-Based Enumeration and Non-Destructive Sizing (LENS)

When performing non-destructive sizing and counting of particles withina container, in some embodiments, there are appreciable artifactsgenerated by the container itself. The liquid interface refracts thelight passing through the vial, which causes appreciable distortions inthe image or images of the particles used for the sizing and countingprocedure. As a result, particles of a given size appear up to, e.g.,four times as large in the image, depending on their spatial positionwithin the vial. Note that for a cylindrical container, the particleimage is typically only stretched along the lateral axis, not thevertical axis of the vial. (See FIG. 5E for an illustration of theseeffects).

As noted above, in some embodiments, these distortion effects may becorrected (e.g., mitigated or even eliminated) using corrective opticaltechniques. However, in some embodiments, such optical correction may beincomplete or unavailable. In such cases, one cannot perform a directcorrelation of the size of a particle to the corresponding image on thedetector.

For example, FIG. 32 shows a histogram for the detected image size for apopulation of standard sized (as shown 100 μm diameter) particles(polymer microspheres) in a fluid acquired using a system wheredistortion from the container has not been corrected (corresponding tothe situation shown in FIG. 5E). A significant variation in apparentimage sizes due to container distortion effects is clearly shown.

This variation makes differentiation between populations of particles ofdifferent sizes difficult, as there may be a substantial overlap in theapparent area on the detector from each size population. For exampleFIG. 33 shows histograms for the detected image size for two populationof standard sized (as shown 100 μm and 140 μm diameter) particles in afluid. Significant overlap between the histograms for the two sizepopulations is clearly shown.

In some embodiments, a processing technique may be applied to recoveraccurate sizing information even in the presence of the distortioneffect described above. The processing is calibrated using data obtainedusing known size standards. For example, FIG. 34 shows experimentallyacquired apparent size histograms for four different populations ofstandard size particles (polymer microspheres). Although fourcalibration curves are shown, in various embodiments, any suitablenumber may be used. In some embodiments, at least two, at least three,at least four, at least five, or at least six curves may be used. Insome embodiments, the number of curves is in the range of 2-100, or anysubrange thereof, such as 4-6. In some embodiments, a set ofexperimental calibration curves can be interpolated to generateadditional curves (e.g., corresponding to size values between theexperimentally measured values).

In some embodiments, the calibration curves may correspond to particlepopulations having actual sizes that differ by any suitable amount,e.g., at least 1 μm, at least 5 μm, at least 10 μm, at least 20 μm, ormore, e.g., in the range of 1 μm to 1000 μm or any subrange thereof.

Once the calibration curves have been determined, the apparent sizedistribution curve for a sample with particles having unknown sized maybe obtained (e.g., from a static image or images, or any other suitabletechnique). The sample curve may be obtained under the same or similarexperimental conditions (e.g., the same or similar container size andshape, fluid properties, illumination conditions, imaging conditions,etc.), This sample curve is compared to the calibration curves todetermine information indicative of the sizes of the particles in thesample.

For example, in some embodiments, a weighted superposition of thecalibration curves is compared to the sample curve. The weighting of thesuperposition is varied to fit the superposition to the sample curve,e.g., using any suitable fitting techniques known in the art. Theweighting of the best fit to the sample curve is then providesinformation about the actual sizes of the particle in the sample. Forexample, in some embodiments, the number of times each calibration curveappears in the best fit superposition corresponds to the count of thatsize species within the sample.

FIG. 35 illustrates the fitting of a superposition of calibration curvesto an experimental sample curve. In this case, the sample was preparedsuch that it was known that the particles were within the range of75-125 μm in diameter. FIG. 36 shows the resulting size counts from thefit, compared with size counts obtained by simply binning the rawapparent size from the corresponding image. For the raw data, there aresignificant numbers of spurious counts outside the actual 75-125 μm sizerange. In contrast, the results obtained from the fit of the calibrationcurves show a greatly reduced number of spurious counts.

Note that, although one possible approach to comparing the sample datato the calibration data has been described, other suitable techniquesmay be used. For example, in some embodiments, the sample curve may bedecomposed using the calibration curves as basis functions, akin to theFourier decomposition of a waveform using sinusoidal basis functions. Ingeneral any suitable convolution, deconvolution, decomposition, or othertechnique may be used.

In some embodiments, the Light-Based Enumeration and Non-Destructive(“LENS”) sizing techniques may be used in combination with the particletracking techniques as previously described. For example, the LENStechnique will tend to operate better when the particles' shapeapproximates that of particles in the size standards used to generatethe calibration data. Additionally, the techniques tend to perform wellwhen the number of particles is high (e.g. greater than 10, greater than50, greater than 100, or more), providing a larger data set for thealgorithm to process

However, in some applications, the number of particles present may below. In some applications, the focus may be on the larger particles inthe sample. Further, in some applications, the sample may includeparticles having shapes that differ from that of the size standardparticles. For example fibers would be elongated rather than thespherical shape used in may standards. Under these conditions, the LENStechniques may not work effectively.

In general any number of particles may be counted using the techniquesdescribed above. In some embodiments, an upper limit on the number ofparticles that may be counted is determined by particle/particle overlapin the sample. In general, the more particles present in the container,the more likely it is that two will not appear disjoint to a single 2Ddetector. This is a function of particles per volume and the size of theparticles. Typically, large particles take up more area on the detector(hence more overlap for a given count/ml when compared with smallerparticles). For example, under certain conditions, in an 10 cc vialfilled with 8 ml of fluid, up to about 500 particles with a diameter of50 μm may be counted before undercounting and oversizing effects due toparticle overlap become apparent.

However, the particle tracking techniques presented above may beeffective to counting and sizing relatively large particles.Accordingly, in some embodiments, a hybrid of the two approaches may beused. FIG. 37 shows an exemplary embodiment of such a hybrid process. Instep 3701, an image sequence is recorded, e.g., using any of thetechniques described herein. In step 3702, the image sequence isprocessed (e.g., filtered, thresholded, segmented, etc,). In step 3703particle data produced in step 3702 can be pre-screened for particlesabove a threshold size. These large particles can be removed from thedata set and processed in step 3704 using tracking techniques. This mayprovide quality, time-dependent size measurements of the largeparticles. If there is a background of smaller particles (below the sizethreshold) present, then this can be processed in step 3705 using LENStechniques. The data produced by the two different techniques can thenbe combined into step 3706 to generate a single particle report for thecontainer under scrutiny.

In various embodiments, the size threshold used to determine whichtechnique is applied may be set to any suitable threshold or minimumvalue of about 1 μm or greater, e.g., about in the range of 1-400 μm ofwidth or diameter of particle or any subrange thereof, e.g., about 1 toabout 50 μm, about 50 to about 250 μm, or about 75 to about 100 μm. Insome embodiments the particle data sent to each technique may be chosenusing criteria other than size, e.g., information related to the shapeof the particle. In general, any suitable combination of criteria may beused.

Three Dimensional Imaging and Particle Detection Techniques

As noted above, in some embodiments, automated visual inspection unit100 may include two or more imagers 110, allowing for three dimensionalimaging of the contents of the container 10.

For example FIGS. 38A-38C illustrate a unit 100 featuring three imagers110. As shown, the imagers 110 are located in a circle around thecontainer 10 at 120 degree intervals, however in various embodiments,more or fewer sensors could be employed. The angles between adjacentimaging sensors do not need to be equal to each other, however, in someembodiments, an equal angle arrangement simplifies the image processingtechniques described below.

In some embodiments, each imager 110 is substantially identical. Theimagers 110 may be aligned so that they are all at the same physicalheight in relation to the container 10, with the container 10 centeredin the field of view of each imager.

In some embodiments, even when care is taken to optimize this physicalalignment, small errors in placement may occur. To account for this, theimagers 110 may be calibrated by imaging a known calibration fixture.Any sufficiently small lateral or vertical alignment deviations can thenbe accounted for by re-sampling and shifting the captured imagesaccordingly. In some embodiments, the images may be processed to correctfor variations in sensitivity or other performance characteristicdifferences between the different sensors used in the imagers 110.

FIG. 38C shows a single imaging arm for the unit 100. As described indetail above, by employing a telecentric imaging arrangement, oneassures that only rays substantially parallel to the imaging axis reachthe sensor surface of the imager 110. As shown in FIG. 39, usinggeometric ray optics techniques (or other suitable techniques), one canestablish a model of the rays inside the container 10 that wouldpropagate through the container wall and reach the sensor surface.

With the ray vectors known, one can take a point or region on from thetwo-dimensional image, and propagate that intensity back into thecontainer 10. Taking one horizontal row from the two-dimensional at atime, one can map out a two dimensional horizontal grid within thecontainer volume. The horizontal grids associated with each of the threeimagers 110 may be superimposed to produce a single map. By repeatingthe process for additional horizontal sensor rows, a vertical stack oftwo-dimensional grids can be built up to form a three dimensional (3D)structure, e.g., corresponding to all or part volume of container 10.

Particle candidates may be identified within the resulting 3D structureusing intensity thresholding in a manner similar to that describedabove. Thresholding can be done on the original two-dimensional imagesfrom the imagers 110, or it can be conducted on the horizontal mapswithin the 3D structure after superposition.

Using a thresholded 3D structure, one can identify candidate particlesthereby obtaining a direct measurement of the 3D position of theparticle within the fluid volume of the container 10. In typicalapplications, the 3D position measurement is accurate for most of thefluid volume, however, in some case, e.g., when imagers 110 includetelecentric lenses, one may experience blind spots due to the containercurvature and associated lensing effect (e.g., as shown in FIG. 39,right panel).

When three imaging arms at angles of 120 degrees are used, as shown, theblind spots correlate closely in pairs (see FIG. 39, right panel).Accurate 3D positioning within the three blind spot regions 3901 may beprecluded. However, in those regions, the positional data can beestablished by examining the two dimensional data from the closestimaging arm.

In various embodiments, the blind spot issue can be mitigated oreliminated by increasing the number of sensor arms to ensure overlappingimaging.

Although one example of using multiple imagers 110 to determine 3Dinformation about the contents of the container 10 has been described,it is to be understood that other techniques may be used. For example,in embodiments using two imagers can apply stereoscopic imagingtechniques to determine 3D information.

In some embodiments, e.g. those featuring static or slow moving sample,3D information could be obtained using a rotating imaging arm, in amanner similar to medical computed tomography machines. The rotating armwould acquire a time series of 2D images from various perspectives,which could be used to construct 3D information, e.g., using anysuitable technique, such as those known from medical imaging. If theimages are acquired at a speed that is fast relative to the dynamics ofthe sample, the 3D image may provide accurate 3D information forparticle detection.

In some embodiments, the 3D information generated using the techniquesdescribed above may be suitable for detecting a candidate particleposition, but not ideal for establishing other characteristics of theparticle, e.g., the particle size or shape. Therefore, in someembodiments, a hybrid approach may be used. For example, in someembodiments, the 3D position of a particle is established based on the3D information (e.g., the 3D structure generated as described above).Once three-dimensional positioning of the particles has beenestablished, one can associate with these positions the size and shapemeasurements obtained from two dimensional images from some or all ofthe imagers 110.

In some embodiments, particle tracking can be conducted on the 3Dpositional data, e.g., using 3D tracking techniques similar to twodimensional techniques described above.

In some embodiments 3D tracking provides advantages, particularly whenused in combination with two dimensional images obtained from eachimager 110.

In 3D tracking, particle-particle occlusions (e.g., as shown in FIG. 5E)are reduced or eliminated. In some embodiments, possible occlusions mayoccur, e.g., for dense samples in the blind spots where true 3Dpositioning fails.

As in the two dimensional case described above, in some examples apredictive tracking technique can be used in the 3D context that takeadvantage information related to the fluid dynamics with the container10.

In some embodiments, once 3D particle positions have been tracked,information about characteristics of the particles (e.g., size andshape) can be aggregated from the two dimension data from the multipleimagers 110 into multiple time-dependent data sets for each particle. Insome embodiments, this may allow a more accurate measurement ofindividual particle characteristics (e.g., size and shape) than would bepossible with a single imaging sensor. For example, in some embodiments,this technique allows clearer detection and size measurement ofelongated particles, since the appearance of the particle is no longerdependent strictly on its orientation relative to a single imager 110.

In some embodiments, this approach can be used to mitigate the lensingeffect caused by the curvature of the container 10. Using the 3Dposition of the particle, the measured particle size on the twodimensional images acquired by each of the imagers 110 can be adjustedby to correct for the lensing effect, e.g., by modifying the lateral(horizontal) component of the size measurement with a lensing-effectscaling factor. This scaling factor can be determined based on anoptical model of the propagation of light through the container 10 toeach of the imagers 110, as detailed above.

Spectral Detection

FIG. 45 shows a sensor 4500 (as shown, a grating spectrometer) that maybe used to with a visual inspection unit 100 of the type describedherein. For example, the sensor 4500 may form a fourth imaging arm usedwith the embodiment of the unit 100 shown in FIG. 38A.

The sensor 4500 can be used to detect a characteristic (e.g., a spectralcharacteristic) of one or more particles in the container 10. Forexample, as shown, the container 10 is illuminated with broadband lightsources 122. The sensor 4500 receives light from the container 10through distortion corrective optics 4501 (e.g., of any of the typesdescribed above), and a telecentric lens 4501. The light from the lens4501 is directed onto a diffraction grating 4503, that separates thespectral components of the light, which are then imaged on an imagingsensor 4504. In some embodiments, the diffraction grating 4503 operatessuch that the position of the incident light along one dimension of thesensor 4504 (e.g., the vertical dimension) corresponds to the wavelengthof the light. The other dimension on the imaging sensor 4504 correspondsto different spatial positions within the container 10. That is, thesensor 4500 provides spectral information for a sub-region of thecontainer, e.g., in the configuration show a the sub-region is ahorizontal “slice” of the container 10.

As particles pass through this central, horizontal plane, theirspectroscopic signature can be recorded. At the same time, as describedin detail above, the conventional imaging arms of the unit 100 may beused to track the position of the particle within the container (e.g.,in three dimensions). This information can be used to determine when agiven particle enters the detection sub-region covered by the sensor4500. When the particle enters the sub-region, the sensor 4500 willsense a characteristic (e.g. a spectral signature) of the particle. Theunit 100 can generate data related to this characteristic, and associatethis data with data indicative of the identity of the particle in thetracking data.

In various embodiments, the characteristic data can be used for anysuitable purpose, e.g., identifying the particle type. For example,spectral information about a given particle can be combined with size,shape, movement or other information about the particle in order todetermine the type of the particle.

In some embodiments, the sensor 4500 and illuminating light sources 122may be modified to detect particle fluorescence, or any other suitablecharacteristics. In general, any spectral characteristic of theparticles may be detected, including a color, an absorption spectrum, anemission spectrum, or a transmission spectrum or a combination of any ofthese.

Although in the example described above, the sensor 4500 is included ina unit 100 featuring three image arms, in other embodiments any othersuitable number of imaging arms may be used, e.g., one, two, four, five,or more. In some embodiments where a single imaging arm is used, thesensor 4500 may be aligned with the imaging arm, e.g., by using a beamsplitter (not shown) to split a beam of light from the container 10, anddirect components to the single imaging arm and the sensor 4500. Inother embodiments, e.g., where multiple imaging arms are used, thesensor 4500 may be oriented at any suitable angle relative to theimagers.

In-Situ Measurements of Sample Properties

In some embodiments, the inspection unit 100 may include one or moredetectors (not shown) that may be used to measure the refractive indexof the fluid in the container 10. For example, in some embodiments, anarrow off-axis collimated laser beam may be directed through a fluidfilled portion of the container 10 and detected to measure thedisplacement of the beam due to refraction through the container 10. Ifthe material and geometry of the container 10 is known, this informationmay be used to determine the refractive index of the fluid. In variousembodiments, any other suitable index of refraction detection techniquemay be used.

In some embodiments, the measured refractive index of the fluid may beused as an input parameter in any of the processing schemes describedherein (e.g., processing used to compensate for lensing effects causedby the curvature of the container 10).

In some embodiments, the inspection unit 100 may also include one ormore detectors (not shown) that may be used to measure informationindicative the shape of the container 10. For example, in someembodiments, a narrow off-axis collimated laser beam may be directedthrough an air filled (e.g., upper) portion of the container 10 anddetected to measure the displacement of the beam relative to areference. The deflection may be used to precisely measure the thicknessof the wall of the container (e.g., with an accuracy of 0.25 mm orless). In various embodiments, any other suitable technique fordetermining the shape of the container may be used.

In some embodiments, the detected geometric information may be used,e.g., as described above, in determining the refractive index of thefluid in the container 10. In some embodiments, the detected geometricinformation may be used as an input parameter for various processingtechniques described herein (e.g., processing used to compensate forlensing effects caused by the curvature of the container 10), or anyother suitable purpose.

Immersion Imaging

As discussed in detail herein, in various embodiments the refractiveproperties of the fluid in container 10 may cause unwanted imagedistortion effects. In some embodiments, these effects may be mitigatedby filing some or all of the space between the container 10 and animager 110 used to image the container with a medium that has an indexof refraction that more closely matches the index of the fluid than air.

In some embodiments, refractive distortion may be further mitigated bymatching the refractive index of the container 10 the fluid containedwithin the container.

In some embodiments, these immersion imaging techniques may reduce oreliminate the need for corrective optics and or processing used toreduce distortion (e.g., the lensing effect described in detail above).

Sample Temperature Control

In some embodiments, the inspection unit 100 may include one or moredevices (not shown) for controlling the temperature of the sample withinthe container 10. For example, in some embodiments, the temperaturecontrol device may be used to vary the temperature of the containerwithin the range of ° C. to 40° C., 0° C. to 100° C., or other suitableranges. In some embodiments, the temperature control device may maintainthe temperature at a stable value, e.g. a value that varies by less than5° C., 2.5° C., 1° C., 0.1° C., 0.01° C., or less.

Temperature control may be particularly advantageous in applicationswhere temperature control is important for ensuring that the samples donot deteriorate during the detection process. In some embodiments, byvarying the temperature of the sample in a controlled manner,temperature and time-dependent stability studies may be conducted fortemperature sensitive products. For example, the platform could be usedto measure the dissolution (or in some cases, formation) of proteinaggregates as drug product is controllably increased in temperaturefrom, e.g., 4° C. (refrigeration) to 20° C. (room temperature), or to37° C. (human body temperature).

In various embodiments, temperature control may be accomplished usingany suitable technique. In some embodiments, the environment within theinspection unit may be sealed and thermally isolated, and thetemperature controlled using, e.g., an air conditioning unit. In someembodiments, a heating coil and a thermoelectric cooler (e.g., a Peltiercooler) may be integrated in a sample holder for the container 10. Inembodiments where multiple containers are held in a tray, thetemperature of the tray may be controlled by circulated aheating/cooling working fluid through the tray (e.g., by passing theworking fluid through a heat exchanger). In general one or moretemperature sensors and or thermostats may be used to provide closedloop temperature control.

Iterative Inspection Techniques

In some embodiments, the inspection unit 100 may re-run the inspectionof a given sample with one or more modified operating parameters (e.g.,spin speed) that may be chosen based on the outcome of an initialinspection run. This process may be repeated iteratively to betteradjust the operating parameter to the particular sample under inspection

For example, in some embodiments, the inspection can be re-run (e.g.,with a modified spin speed) if the output of a particle detection andsizing operation returns results that are outside a range of expectedresults (indicating an error in the initial inspection).

Background Reference Mapping for Auto-Calibration

As described in detail above, in various embodiments it is desirable tocharacterize distortion effects (e.g., lensing effects) caused byrefraction of light passing through the container 10 to an imager 110.In some embodiments, the inspection unit 100 itself may be used to mapout the distortions caused by the container 10. This map can then beused during image processing to compensate for these effects.

For example, in some embodiments, one or more calibration indicia (e.g.,a grid) may be placed behind the container 10 as a background for animager 110. By detecting these indicia in the acquired image (e.g.,using edge detection or other suitable feature detection techniques),and comparing their appearance to the known actual appearance, therefractive distortion may be detected and mapped.

In some embodiments, this approach may be used to correct for distortioncaused by non-cylindrical containers, e.g., containers that arerotationally symmetric about an axis, but with varying circumferencesabout the axis (such as containers having shapes familiar fromcontemporary plastic soda bottles).

Vibration Auto Detection and Mitigation

As noted above, in some embodiments, vibrations can degrade theperformance of the inspection unit 100. Vibrations cause otherwisestatic features (such as cosmetic defects on the container surface) tooscillate during video acquisition. This may reduce the performance ofthe static feature removal phase, by creating small but significantoscillating halos that survive the static feature removal andpotentially cause spurious results in subsequent particle detectionalgorithms. In various embodiments, one or more of the followingtechniques may be used to reduce the effect of vibration.

In some embodiments, the oscillating halo features that form aroundremoved static features can be mitigated by increasing the size of imageregion corresponding to the detected static features (e.g., by athickness of one or several pixels) so that the areas of the imagecontaining the thin oscillating halos are also deleted prior to theparticle analysis phase. However, in some embodiments, this approach maybe disadvantageous in that it serves to reduce the effective availablesensor area.

In some embodiments, a screening algorithm to detect the presence of theoscillating features. For example, the features may be detected byprocessing the image to locate features that oscillate, but do nottranslate across the image. In some embodiments, the features can befurther identified based on their proximity to detected static features.

In some embodiments, characteristics of the vibration of the containermay be detected from the captured images, e.g., using edge detection todetect movement of the container walls, so that the system canautomatically detect and potentially warn users of unacceptably highlevels of environmental vibrations.

In some embodiments, characteristics of the vibration of the containermay be detected using physical sensors. For example, in someembodiments, a tool head holding and manipulating the container duringinspection may include motion detection devices (e.g., high-precisionaccelerometers) which provide feedback from which the system canautomatically provide warning to users regarding vibration levels abovean established threshold.

Examples

The following provides exemplary performance characteristics for anembodiment of and automated visual inspection unit 100 of the typedescribed herein.

Referring to FIG. 40, the unit 100 was presented with containers 10 eachincluding only a single polymer sphere of a known size. Multipledetection runs (n=80) were performed on each container and the detectionpercentage measured (data bars labeled “APT” in the figure). As shown,the detection percentage for the system was above 90% for particle sizesranging from 15-200 μm in diameter. Detection percentages for the sametask performed visually by a trained human are presented for comparison(data bars labeled “human”). Note that human detection capability fallsoff rapidly for particle sized below 200

Referring to FIG. 41, in another test, the unit 100 was presented withcontainers holding particles above and below the visible cutoff of 125μm in diameter. The unit 100 detected the particle and also classifiedthe particle based on size as being above or below the visible cutoff of125 μm. As shown, the detection percentage for the system was above 90%for particle sizes ranging from 15-200 μm in diameter. The unit 100 alsocorrectly categorized the detected particles with a very high degree ofaccuracy.

Referring to FIG. 42 dilution series were created for multiple sizestandards, each series made up of containers holding particles at agiven concentration. The resulting containers were analyzed by the unit100 to provide a particle count, and regression was used to determineR-square “R̂2” values for linearity of count versus inverse dilutionfactor. As shown, the “R̂2” value was above 0.95 for particle sizesranging from 15-200 μm, indicating excellent linearity.

Referring to FIG. 43, a stressed sample containing protein particles wasanalyzed by the unit 100 to determine a particle count binned byparticle size. The precision of the particle count for each bin takenover 10 runs is shown. The protein particles are of unknown size, whichmakes absolute size accuracy comparison impossible, however, as shown,the precision of the system for counting and sizing the proteins ishigh. The normalized error for the measurement was 3%, indicatingexcellent precision.

Referring to FIG. 44, the unit 100 was also characterized at detectingblank vs. protein particle containing vials. The performance of the unit100 was compared with that of a certified visual inspector observing thesame set of vials. The unit 100 (labeled “APT” in the figure) detectedall 40 protein vials and 80 blanks correctly in triplicate runs. Theself agreement at classifying visible and subvisible particles was 100%.Humans scored only around 85% in both categories.

CONCLUSION

Those of ordinary skill in the art realize that processes involved in anautomated system and method for nondestructive particle detection andidentification (processing time-series data acquired through visualinspection) may be embodied in an article of manufacture that includes acomputer-usable medium. For example, such a computer usable medium caninclude a readable memory device, such as a hard drive device, a CD-ROM,a DVD-ROM, a computer diskette or solid-state memory components (ROM,RAM), having computer readable program code segments stored thereon. Thecomputer readable medium can also include a communications ortransmission medium, such as a bus or a communications link, eitheroptical, wired, or wireless, having program code segments carriedthereon as digital or analog data signals.

A flow diagram is used herein. The use of flow diagrams is not meant tobe limiting with respect to the order of operations performed. Theherein described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely exemplary, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (e.g., bodies of theappended claims) are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations.

However, the use of such phrases should not be construed to imply thatthe introduction of a claim recitation by the indefinite articles “a” or“an” limits any particular claim containing such introduced claimrecitation to subject mater containing only one such recitation, evenwhen the same claim includes the introductory phrases “one or more” or“at least one” and indefinite articles such as “a” or “an” (e.g., “a”and/or “an” should typically be interpreted to mean “at least one” or“one or more”); the same holds true for the use of definite articlesused to introduce claim recitations. In addition, even if a specificnumber of an introduced claim recitation is explicitly recited, thoseskilled in the art will recognize that such recitation should typicallybe interpreted to mean at least the recited number (e.g., the barerecitation of “two recitations,” without other modifiers, typicallymeans at least two recitations, or two or more recitations).

Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (e.g., “a system having at least one of A, B, and C”would include but not be limited to systems that have A alone, B alone,C alone, A and B together, A and C together, B and C together, and/or A,B, and C together, etc.). In those instances where a conventionanalogous to “at least one of A, B, or C, etc.” is used, in general sucha construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, or C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.).

It will be further understood by those within the art that virtually anydisjunctive word and/or phrase presenting two or more alternative terms,whether in the description, claims, or drawings, should be understood tocontemplate the possibilities of including one of the terms, either ofthe terms, or both terms. For example, the phrase “A or B” will beunderstood to include the possibilities of “A” or “B” or “A and B.”

As used herein, the term optical element may refer to one or morerefractive, reflective, diffractive, holographic, polarizing, orfiltering elements in any suitable combination. As used herein termssuch as “light”, “optical”, or other related terms should be understoodto refer not only to light visible to the human eye, but may alsoinclude, for example, light in the ultraviolet, visible, and infraredportions of the electromagnetic spectrum.

The foregoing description of illustrative embodiments has been presentedfor purposes of illustration and of description. It is not intended tobe exhaustive or limiting with respect to the precise form disclosed,and modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the disclosed embodiments.It is intended that the scope of the invention be defined by the claimsappended hereto and their equivalents.

What is claimed is:
 1. A method of nondestructive counting and sizing ofundissolved particles in a vessel that is at least partially filled witha fluid, the method comprising: (a) receiving at least one image of theparticles in the vessel obtained under specified imaging conditions; (b)based on the at least one image, detecting the particles and determininginformation indicative of the apparent size of the detected particles inthe image; (c) determining apparent particle size population informationindicative of an apparent particle size distribution of the detectedparticles; and (d) determining actual particle size populationinformation indicative of the actual particle size distribution of thedetected particles based on (i) the apparent particle size populationinformation and (ii) calibration population information indicative ofthe apparent size distribution of one or more sets of standard sizedparticles imaged under conditions corresponding to the specified imagingconditions.
 2. The method of claim 1, wherein step (d) comprises:fitting a superposition of apparent size distributions for a pluralityof the sets of standard sized particles to the apparent particle sizepopulation of the detected particles.
 3. The method of claim 2, whereinthe plurality of sets of standard sized particles comprises at leastfour sets each having a different respective standard size.
 4. Themethod of claim 3, wherein the respective size of each one of the atleast four sets of standard sized particles differs by at least 1 μmfrom the respective size of each of the other ones of the at least foursets of standard sized particles.
 5. An apparatus for counting andsizing undissolved particles in a vessel that is at least partiallyfilled with a fluid, the apparatus comprising at least one processorconfigured to: (a) receive at least one image of the particles in thevessel obtained under specified imaging conditions; (b) based on the atleast one image, detect the particles and determine informationindicative of the apparent size of the detected particles in the image;(c) determine apparent particle size population information indicativeof an apparent particle size distribution of the detected particles; and(d) determine actual particle size population information indicative ofthe actual particle size distribution of the detected particles based on(i) the apparent particle size population information and (ii)calibration population information indicative of the apparent sizedistribution of one or more sets of standard sized particles imagedunder conditions corresponding to the specified imaging conditions. 6.The apparatus of claim 5, wherein the processor is configured to: fit asuperposition of apparent size distributions for a plurality of sets ofstandard sized particles imaged to the apparent particle size populationof the detected particles.
 7. The apparatus of claim 6, wherein theplurality of sets of standard sized particles comprises at least foursets each having a different respective standard size.
 8. The apparatusof claim 7, wherein the respective size of each one of the at least foursets of standard sized particles differs by at least 1 μm from therespective size of each of the other ones of the at least four sets ofstandard sized particles.
 9. A computer program product fornondestructive counting and sizing of undissolved particles in a vesselthat is at least partially filled with a fluid, the computer programproduct comprising nonvolatile, machine-readable instructions, which,when executed by a processor, cause the processor to (a) receiving atleast one image of the particles in the vessel obtained under specifiedimaging conditions; (b) based on the at least one image, detecting theparticles and determining information indicative of the apparent size ofthe detected particles in the image (c) determining apparent particlesize population information indicative of an apparent particle sizedistribution of the detected particles; and (d) determining actualparticle size population information indicative of the actual particlesize distribution of the detected particles based on (i) the apparentparticle size population information and (ii) calibration populationinformation indicative of the apparent size distribution of one or moresets of standard sized particles imaged under conditions correspondingto the specified imaging conditions.